Sciweavers

Share
SIGIR
2010
ACM

Retrieval system evaluation: automatic evaluation versus incomplete judgments

12 years 10 months ago
Retrieval system evaluation: automatic evaluation versus incomplete judgments
In information retrieval (IR), research aiming to reduce the cost of retrieval system evaluations has been conducted along two lines: (i) the evaluation of IR systems with reduced (i.e. incomplete) amounts of manual relevance assessments, and (ii) the fully automatic evaluation of IR systems, thus foregoing the need for manual assessments altogether. The proposed methods in both areas are commonly evaluated by comparing their performance estimates for a set of systems to a ground truth (provided for instance by evaluating the set of systems according to mean average precision). In contrast, in this poster we compare an automatic system evaluation approach directly to two evaluations based on incomplete manual relevance assessments. For the particular case of TREC's Million Query track, we show that the automatic evaluation leads to results which are highly correlated to those achieved by approaches relying on incomplete manual judgments. Categories and Subject Descriptors: H.3.3 ...
Claudia Hauff, Franciska de Jong
Added 06 Dec 2010
Updated 06 Dec 2010
Type Conference
Year 2010
Where SIGIR
Authors Claudia Hauff, Franciska de Jong
Comments (0)
books