Sciweavers

Share
SIGIR
2010
ACM

Human performance and retrieval precision revisited

10 years 8 months ago
Human performance and retrieval precision revisited
Several studies have found that the Cranfield approach to evaluation can report significant performance differences between retrieval systems for which little to no performance difference is found for humans completing tasks with these systems. We revisit the relationship between precision and performance by measuring human performance on tightly controlled search tasks and with user interfaces offering limited interaction. We find that human performance and retrieval precision are strongly related. We also find that users change their relevance judging behavior based on the precision of the results. This change in behavior coupled with the well-known lack of perfect inter-assessor agreement can reduce the measured performance gains predicted by increased precision. Categories and Subject Descriptors: H.3.3 [Information Search and Retrieval] General Terms: Experimentation, Human Factors, Performance
Mark D. Smucker, Chandra Prakash Jethani
Added 16 Aug 2010
Updated 16 Aug 2010
Type Conference
Year 2010
Where SIGIR
Authors Mark D. Smucker, Chandra Prakash Jethani
Comments (0)
books