Investigating the relationship between language model perplexity and IR precision-recall measures

An empirical study has been conducted investigating the relationship between the performance of a generative language model in terms of perplexity and the corresponding information retrieval performance obtained. It is observed, on the corpora considered, that the perplexity of the language model has a systematic relationship with the achievable precision recall performance. Categories and Subject Descriptors I.2.7 [Artificial Intelligence]: Natural Language Processing—Language Models; H.3.3 [Information Storage and Retrieval]: Information Search and Retrieval—Retrieval Models General Terms Algorithms, Experimentation

• Type: Conference
• Year: 2003
• Where: SIGIR
• Authors: Leif Azzopardi, Mark Girolami, Keith van Rijsbergen
• Keywords: Generative Language Model, Information Retrieval Performance, Language Model, SIGIR 2003