Retrieval evaluation with incomplete information

9 years 2 months ago
Retrieval evaluation with incomplete information
This paper examines whether the Cranfield evaluation methodology is robust to gross violations of the completeness assumption (i.e., the assumption that all relevant documents within a test collection have been identified and are present in the collection). We show that current evaluation measures are not robust to substantially incomplete relevance judgments. A new measure is introduced that is both highly correlated with existing measures when complete judgments are available and more robust to incomplete judgment sets. This finding suggests that substantially larger or dynamic test collections built using current pooling practices should be viable laboratory tools, despite the fact that the relevance information will be incomplete and imperfect. Categories and Subject Descriptors H.3.4 [Information Storage and Retrieval]: Systems and Software—Performance evaluation General Terms Measurement,Experimentation Keywords Cranfield, incomplete judgments
Chris Buckley, Ellen M. Voorhees
Added 30 Jun 2010
Updated 30 Jun 2010
Type Conference
Year 2004
Authors Chris Buckley, Ellen M. Voorhees
Comments (0)