Sciweavers

Share
CIKM
2009
Springer

Improvements that don't add up: ad-hoc retrieval results since 1998

11 years 11 months ago
Improvements that don't add up: ad-hoc retrieval results since 1998
The existence and use of standard test collections in information retrieval experimentation allows results to be compared between research groups and over time. Such comparisons, however, are rarely made. Most researchers only report results from their own experiments, a practice that allows lack of overall improvement to go unnoticed. In this paper, we analyze results achieved on the TREC Ad-Hoc, Web, Terabyte, and Robust collections as reported in SIGIR (1998–2008) and CIKM (2004–2008). Dozens of individual published experiments report effectiveness improvements, and often claim statistical significance. However, there is little evidence of improvement in ad-hoc retrieval technology over the past decade. Baselines are generally weak, often being below the median original TREC system. And in only a handful of experiments is the score of the best TREC automatic run exceeded. Given this finding, we question the value of achieving even a statistically significant result over a we...
Timothy G. Armstrong, Alistair Moffat, William Web
Added 24 Jul 2010
Updated 24 Jul 2010
Type Conference
Year 2009
Where CIKM
Authors Timothy G. Armstrong, Alistair Moffat, William Webber, Justin Zobel
Comments (0)
books