Term necessity prediction

10 years 5 months ago
Term necessity prediction
The probability that a term appears in relevant documents ( ) is a fundamental quantity in several probabilistic retrieval models, however it is difficult to estimate without relevance judgments or a relevance model. We call this value term necessity because it measures the percentage of relevant documents retrieved by the term – how necessary a term‟s occurrence is to document relevance. Prior research typically either set this probability to a constant, or estimated it based on the term's inverse document frequency, neither of which was very effective. This paper identifies several factors that affect term necessity, ple, a term‟s topic centrality, synonymy and abstractness. It develops term- and query-dependent features for each factor that enable supervised learning of a predictive model of term necessity from training data. Experiments with two popular retrieval models and 6 standard datasets demonstrate that using predicted term necessity estimates as user term weight...
Le Zhao, Jamie Callan
Added 24 Jan 2011
Updated 24 Jan 2011
Type Journal
Year 2010
Where CIKM
Authors Le Zhao, Jamie Callan
Comments (0)