Measuring ineffectiveness

12 years 11 months ago
Measuring ineffectiveness
An evaluation methodology that targets ineffective topics is needed to support research on obtaining more consistent retrieval across topics. Using average values of traditional evaluation measures is not an appropriate methodology because it emphasizes effective topics: poorly performing topics’ scores are by definition small, and they are therefore difficult to distinguish from the noise inherent in retrieval evaluation. We examine two new measures that emphasize a system’s worst topics. While these measures focus on different aspects of retrieval behavior than traditional measures, the measures are less stable than traditional measures and the margin of error associated with the new measures is large relative to the observed differences in scores. Categories and Subject Descriptors H.3.4 [Information Storage and Retrieval]: Systems and Software—Performance evaluation General Terms Measurement, Experimentation Keywords evaluation, worst-case behavior
Ellen M. Voorhees
Added 30 Jun 2010
Updated 30 Jun 2010
Type Conference
Year 2004
Authors Ellen M. Voorhees
Comments (0)