Sciweavers

CIKM
2006
Springer

A document-centric approach to static index pruning in text retrieval systems

13 years 7 months ago
A document-centric approach to static index pruning in text retrieval systems
We present a static index pruning method, to be used in ad-hoc document retrieval tasks, that follows a documentcentric approach to decide whether a posting for a given term should remain in the index or not. The decision is made based on the term's contribution to the document's Kullback-Leibler divergence from the text collection's global language model. Our technique can be used to decrease the size of the index by over 90%, at only a minor decrease in retrieval effectiveness. It thus allows us to make the index small enough to fit entirely into the main memory of a single PC, even for large text collections containing millions of documents. This results in great efficiency gains, superior to those of earlier pruning methods, and an average response time around 20 ms on the GOV2 document collection. Categories and Subject Descriptors H.2.4 [Systems]: Textual databases; H.3.1 [Content Analysis and Indexing]: Indexing methods General Terms Experimentation, Performance ...
Stefan Büttcher, Charles L. A. Clarke
Added 20 Aug 2010
Updated 20 Aug 2010
Type Conference
Year 2006
Where CIKM
Authors Stefan Büttcher, Charles L. A. Clarke
Comments (0)