Forming test collections with no system pooling

9 years 2 months ago
Forming test collections with no system pooling
Forming test collection relevance judgments from the pooled output of multiple retrieval systems has become the standard process for creating resources such as the TREC, CLEF, and NTCIR test collections. This paper presents a series of experiments examining three different ways of building test collections where no system pooling is used. First, a collection formation technique combining manual feedback and multiple systems is adapted to work with a single retrieval system. Second, an existing method based on pooling the output of multiple manual searches is re-examined: testing a wider range of searchers and retrieval systems than has been examined before. Third, a new approach is explored where the ranked output of a single automatic search on a single retrieval system is assessed for relevance: no pooling whatsoever. Using established techniques for evaluating the quality of relevance judgments, in all three cases, test collections are formed that are as good as TREC. Categories an...
Mark Sanderson, Hideo Joho
Added 30 Jun 2010
Updated 30 Jun 2010
Type Conference
Year 2004
Authors Mark Sanderson, Hideo Joho
Comments (0)