Understandability Biased Evaluation for Information Retrieval

4 years 3 months ago
Understandability Biased Evaluation for Information Retrieval
Abstract. Although relevance is known to be a multidimensional concept, information retrieval measures mainly consider one dimension of relevance: topicality. In this paper we propose a method to integrate multiple dimensions of relevance in the evaluation of information retrieval systems. This is done within the gain-discount evaluation framework, which underlies measures like rank-biased precision (RBP), cumulative gain, and expected reciprocal rank. Albeit the proposal is general and applicable to any dimension of relevance, we study specific instantiations of the approach in the context of evaluating retrieval systems with respect to both the topicality and the understandability of retrieved documents. This leads to the formulation of understandability biased evaluation measures based on RBP. We study these measures using both simulated experiments and real human assessments. The findings show that considering both understandability and topicality in the evaluation of retrieval s...
Guido Zuccon
Added 02 Apr 2016
Updated 02 Apr 2016
Type Journal
Year 2016
Where ECIR
Authors Guido Zuccon
Comments (0)