Storing the Web in Memory: Space Efficient Language Models with Constant Time Retrieval

We present three novel methods of compactly storing very large n-gram language models. These methods use substantially less space than all known approaches and allow n-gram probabilities or counts to be retrieved in constant time, at speeds comparable to modern language modeling toolkits. Our basic approach generates an explicit minimal perfect hash function, that maps all n-grams in a model to distinct integers to enable storage of associated values. Extensions of this approach exploit distributional characteristics of n-gram data to reduce storage costs, including variable length coding of values and the use of tiered structures that partition the data for more efficient storage. We apply our approach to storing the full Google Web1T n-gram set and all 1-to-5 grams of the Gigaword newswire cor

• Type: Journal
• Year: 2010
• Where: EMNLP
• Authors: David Guthrie, Mark Hepple
• Keywords: EMNLP 2010, Google Web1T N-gram, N-gram, N-gram Language Models, Natural Language Processing