7 search results - page 1 / 2 » Relevance judgments between TREC and Non-TREC assessors |
SIGIR
2008
ACM
13 years 4 months ago
2008
ACM
This paper investigates the agreement of relevance assessments between official TREC judgments and those generated from an interactive IR experiment. Results show that 63% of docu...
SIGIR
2012
ACM
11 years 7 months ago
2012
ACM
Assessors frequently disagree on the topical relevance of documents. How much of this disagreement is due to ambiguity in assessment instructions? We have two assessors assess TRE...
SIGIR
2008
ACM
13 years 4 months ago
2008
ACM
Information retrieval evaluation has typically been performed over several dozen queries, each judged to near-completeness. There has been a great deal of recent work on evaluatio...
SIGIR
2008
ACM
13 years 4 months ago
2008
ACM
There has been recent interest in collecting user or assessor preferences, rather than absolute judgments of relevance, for the evaluation or learning of ranking algorithms. Since...
CIKM
2007
Springer
13 years 8 months ago
2007
Springer
Information retrieval experimentation generally proceeds in a cycle of development, evaluation, and hypothesis testing. Ideally, the evaluation and testing phases should be short ...