Time-based calibration of effectiveness measures

9 years 2 months ago
Time-based calibration of effectiveness measures
Many current effectiveness measures incorporate simplifying assumptions about user behavior. These assumptions prevent the measures from reflecting aspects of the search process that directly impact the quality of retrieval results as experienced by the user. In particular, these measures implicitly model users as working down a list of retrieval results, spending equal time assessing each document. In reality, even a careful user, intending to identify as much relevant material as possible, must spend longer on some documents than on others. Aspects such as document length, duplicates and summaries all influence the time required. In this paper, we introduce a time-biased gain measure, which explicitly accommodates such aspects of the search process. By conducting an appropriate user study, we calibrate and validate the measure against the TREC 2005 Robust Track test collection. We examine properties of the measure, contrasting it to traditional effectiveness measures, and explor...
Mark D. Smucker, Charles L. A. Clarke
Added 28 Sep 2012
Updated 28 Sep 2012
Type Journal
Year 2012
Authors Mark D. Smucker, Charles L. A. Clarke
Comments (0)