Here or There

13 years 1 months ago
Here or There
Abstract. Information retrieval systems have traditionally been evaluated over absolute judgments of relevance: each document is judged for relevance on its own, independent of other documents that may be on topic. We hypothesize that preference judgments of the form "document A is more relevant than document B" are easier for assessors to make than absolute judgments, and provide evidence for our hypothesis through a study with assessors. We then investigate methods to evaluate search engines using preference judgments. Furthermore, we show that by using inferences and clever selection of pairs to judge, we need not compare all pairs of documents in order to apply evaluation methods.
Ben Carterette, Paul N. Bennett, David Maxwell Chi
Added 29 Oct 2010
Updated 29 Oct 2010
Type Conference
Year 2008
Where ECIR
Authors Ben Carterette, Paul N. Bennett, David Maxwell Chickering, Susan T. Dumais
Comments (0)