Very Predictive Ngrams for Space-Limited Probabilistic Models

In sequential prediction tasks, one repeatedly tries to predict the next element in a sequence. A classical way to solve these problems is to fit an order-n Markov model to the data, but fixed-order models are often bigger than they need to be. In a fixed-order model, all predictors are of length n, even if a shorter predictor would work just as well. We present a greedy algorithm, vpr, for finding variable-length predictive rules. Although vpr is not optimal, we show that on English text, it performs similarly to fixed-order models but uses fewer parameters.

• Type: Conference
• Year: 2003
• Where: IDA
• Authors: Paul R. Cohen, Charles A. Sutton
• Keywords: IDA 2003, Order-n Markov Model, Variable-length Predictive Rules, fixed-order Models