Sciweavers

Share
SIGIR
2010
ACM

Reusable test collections through experimental design

11 years 3 months ago
Reusable test collections through experimental design
Portable, reusable test collections are a vital part of research and development in information retrieval. Reusability is difficult to assess, however. The standard approach— simulating judgment collection when groups of systems are held out, then evaluating those held-out systems—only works when there is a large set of relevance judgments to draw on during the simulation. As test collections adapt to larger and larger corpora, it becomes less and less likely that there will be sufficient judgments for such simulation experiments. Thus we propose a methodology for information retrieval experimentation that collects evidence for or against the reusability of a test collection while judgments are being made. Using this methodology along with the appropriate statistical analyses, researchers will be able to estimate the reusability of their test collections while building them and implement“course corrections”if the collection does not seem to be achieving desired levels of reusa...
Ben Carterette, Evangelos Kanoulas, Virgiliu Pavlu
Added 16 Aug 2010
Updated 16 Aug 2010
Type Conference
Year 2010
Where SIGIR
Authors Ben Carterette, Evangelos Kanoulas, Virgiliu Pavlu, Hui Fang
Comments (0)
books