Probabilistic Document Length Priors for Language Models

8 years 3 months ago
Probabilistic Document Length Priors for Language Models
This paper addresses the issue of devising a new document prior for the language modeling (LM) approach for Information Retrieval. The prior is based on term statistics, derived in a probabilistic fashion and portrays a novel way of considering document length. Furthermore, we developed a new way of combining document length priors with the query likelihood estimation based on the risk of accepting the latter as a score. This prior has been combined with a document retrieval language model that uses Jelinek-Mercer (JM), a smoothing technique which does not take into account document length. The combination of the prior boosts the retrieval performance, so that it outperforms a LM with a document length dependent smoothing component (Dirichlet prior) and other state of the art high-performing scoring function (BM25). Improvements are significant, robust across different collections and query sizes.
Roi Blanco, Alvaro Barreiro
Added 29 Oct 2010
Updated 29 Oct 2010
Type Conference
Year 2008
Where ECIR
Authors Roi Blanco, Alvaro Barreiro
Comments (0)