Sciweavers

SIGIR
2008
ACM

Relevance judgments between TREC and Non-TREC assessors

13 years 3 months ago
Relevance judgments between TREC and Non-TREC assessors
This paper investigates the agreement of relevance assessments between official TREC judgments and those generated from an interactive IR experiment. Results show that 63% of documents judged relevant by our users matched official TREC judgments. Several factors contributed to differences in the agreements: the number of retrieved relevant documents; the number of relevant documents judged; system effectiveness per topic and the ranking of relevant documents. Categories and Subject Descriptors H.3.0 [Information Storage and Retrieval]: General General Terms: Measurement, Human factors
Azzah Al-Maskari, Mark Sanderson, Paul Clough
Added 15 Dec 2010
Updated 15 Dec 2010
Type Journal
Year 2008
Where SIGIR
Authors Azzah Al-Maskari, Mark Sanderson, Paul Clough
Comments (0)