Sciweavers

Share
SIGIR
2012
ACM

An uncertainty-aware query selection model for evaluation of IR systems

9 years 2 months ago
An uncertainty-aware query selection model for evaluation of IR systems
We propose a mathematical framework for query selection as a mechanism for reducing the cost of constructing information retrieval test collections. In particular, our mathematical formulation explicitly models the uncertainty in the retrieval effectiveness metrics that is introduced by the absence of relevance judgments. Since the optimization problem is computationally intractable, we devise an adaptive query selection algorithm, referred to as Adaptive, that provides an approximate solution. Adaptive selects queries iteratively and assumes that no relevance judgments are available for the query under consideration. Once a query is selected, the associated relevance assessments are acquired and then used to aid the selection of subsequent queries. We demonstrate the effectiveness of the algorithm on two TREC test collections as well as a test collection of an online search engine with 1000 queries. Our experimental results show that the queries chosen by Adaptive produce reliable ...
Mehdi Hosseini, Ingemar J. Cox, Natasa Milic-Frayl
Added 28 Sep 2012
Updated 28 Sep 2012
Type Journal
Year 2012
Where SIGIR
Authors Mehdi Hosseini, Ingemar J. Cox, Natasa Milic-Frayling, Milad Shokouhi, Emine Yilmaz
Comments (0)
books