Sciweavers

Share
CIKM
2009
Springer

Empirical justification of the gain and discount function for nDCG

9 years 2 months ago
Empirical justification of the gain and discount function for nDCG
The nDCG measure has proven to be a popular measure of retrieval effectiveness utilizing graded relevance judgments. However, a number of different instantiations of nDCG exist, depending on the arbitrary definition of the gain and discount functions used (1) to dictate the relative value of documents of different relevance grades and (2) to weight the importance of gain values at different ranks, respectively. In this work we discuss how to empirically derive a gain and discount function that optimizes the efficiency or stability of nDCG. First, we describe a variance decomposition analysis framework and an optimization procedure utilized to find the efficiency- or stability-optimal gain and discount functions. Then we use TREC data sets to compare the optimal gain and discount functions to the ones that have appeared in the IR literature with respect to (a) the efficiency of the evaluation, (b) the induced ranking of systems, and (c) the discriminative power of the resulting nDCG me...
Evangelos Kanoulas, Javed A. Aslam
Added 14 Aug 2010
Updated 14 Aug 2010
Type Conference
Year 2009
Where CIKM
Authors Evangelos Kanoulas, Javed A. Aslam
Comments (0)
books