Less is More: Significance-Based N-gram Selection for Smaller, Better Language Models

The recent availability of large corpora for training N-gram language models has shown the utility of models of higher order than just trigrams. In this paper, we investigate methods to control the increase in model size resulting from applying standard methods at higher orders. We introduce significance-based N-gram selection, which not only reduces model size, but also improves perplexity for several smoothing methods, including Katz backoff and absolute discounting. We also show that, when combined with a new smoothing method and a novel variant of weighted-difference pruning, our selection method performs better in the trade-off between model size and perplexity than the best pruning method we found for modified Kneser-Ney smoothing.

• Type: Journal
• Year: 2009
• Where: EMNLP
• Authors: Robert C. Moore, Chris Quirk
• Keywords: EMNLP 2009, Model Size, N-gram Language Models, Natural Language Processing, Smoothing Method