Sciweavers

CIKM
2009
Springer

Relying on topic subsets for system ranking estimation

13 years 10 months ago
Relying on topic subsets for system ranking estimation
Ranking a number of retrieval systems according to their retrieval effectiveness without relying on costly relevance judgments was first explored by Soboroff et al [6]. Over the years, a number of alternative approaches have been proposed. We perform a comprehensive analysis of system ranking estimation approaches on a wide variety of TREC test collections and topics sets. Our analysis reveals that the performance of such approaches is highly dependent upon the topic or topic subset, used for estimation. We hypothesize that the performance of system ranking estimation approaches can be improved by selecting the “right” subset of topics and show that using topic subsets improves the performance by 32% on average, with a maximum improvement of up to 70% in some cases. Categories and Subject Descriptors: H.3.4 Information Storage and Retrieval: Information Search and Retrieval General Terms: Experimentation.
Claudia Hauff, Djoerd Hiemstra, Franciska de Jong,
Added 26 May 2010
Updated 26 May 2010
Type Conference
Year 2009
Where CIKM
Authors Claudia Hauff, Djoerd Hiemstra, Franciska de Jong, Leif Azzopardi
Comments (0)