Sciweavers

Share
SIGIR
2012
ACM

Effect of written instructions on assessor agreement

7 years 9 months ago
Effect of written instructions on assessor agreement
Assessors frequently disagree on the topical relevance of documents. How much of this disagreement is due to ambiguity in assessment instructions? We have two assessors assess TREC Legal Track documents for relevance, some to a general topic description, others to detailed assessment guidelines. We find that detailed guidelines lead to no significant increase in agreement amongst assessors or between assessors and the official qrels. Categories and Subject Descriptors H.3.4 [Information Storage and Retrieval]: Systems and software—performance evaluation. Keywords Retrieval experiment, evaluation, e-discovery General Terms Measurement, performance, experimentation
William Webber, Bryan Toth, Marjorie Desamito
Added 28 Sep 2012
Updated 28 Sep 2012
Type Journal
Year 2012
Where SIGIR
Authors William Webber, Bryan Toth, Marjorie Desamito
Comments (0)
books