Using parsimonious language models on web data

In this paper we explore the use of parsimonious language models for web retrieval. These models are smaller thus more efficient than the standard language models and are therefore well suited for large-scale web retrieval. We have conducted experiments on four TREC topic sets, and found that the parsimonious language model results in improvement of retrieval effectiveness over the standard language model for all data-sets and measures. In all cases the improvement is significant, and more substantial than in earlier experiments on newspaper/newswire data. Categories and Subject Descriptors: H.3 [Information Storage and Retrieval]: H.3.3 Information Search and Retrieval General Terms: Measurement, Experimentation, Performance

• Type: Journal
• Year: 2008
• Where: SIGIR
• Authors: Rianne Kaptein, Rongmei Li, Djoerd Hiemstra, Jaap Kamps
• Keywords: Information Technology, Language Model, Parsimonious Language Model, SIGIR 2008, Standard Language Model