TinyLex: static n-gram index pruning with perfect recall

11 years 7 months ago
TinyLex: static n-gram index pruning with perfect recall
Inverted indexes using sequences of characters (n-grams) as terms provide an error-resilient and language-independent way to query for arbitrary substrings and perform approximate matching in a text, but present a number of practical problems: they have a very large number of terms, they exhibit pathologically expensive worst-case query times on certain natural inputs, and they cannot cope with very short query strings. In word-based indexes, static index pruning has been successful in reducing index size while maintaining precision, at the expense of recall. Taking advantage of the unique inclusion structure of n-gram terms of different lengths, we show that the lexicon size of an n-gram index can be reduced by 7 to 15 times without any loss of recall, and without any increase in either index size or query time. Because the lexicon is typically stored in main memory, this substantially reduces the memory required for queries. Simultaneously, our construction is also the first overlap...
Derrick Coetzee
Added 18 Oct 2010
Updated 18 Oct 2010
Type Conference
Year 2008
Where CIKM
Authors Derrick Coetzee
Comments (0)