Sciweavers

WSDM
2016
ACM

On Obtaining Effort Based Judgements for Information Retrieval

7 years 11 months ago
On Obtaining Effort Based Judgements for Information Retrieval
Document relevance has been the primary focus in the design, optimization and evaluation of retrieval systems. Traditional test collections are constructed by asking judges the relevance grade for a document with respect to an input query. Recent work [44] found an evidence that effort is another important factor in determining document utility, suggesting that more thought should be given into incorporating effort into information retrieval. However, that work did not ask judges to directly assess the level of effort required to consume a document or analyse how effort judgements relate to traditional relevance judgements. In this work, focusing on three aspects associated with effort, we show that it is possible to get judgements of effort from the assessors. We further show that given documents of the same relevance grade, effort needed to find the portion of the document relevant to the query is a significant factor in determining user satisfaction as well as user preferen...
Manisha Verma, Emine Yilmaz, Nick Craswell
Added 12 Apr 2016
Updated 12 Apr 2016
Type Journal
Year 2016
Where WSDM
Authors Manisha Verma, Emine Yilmaz, Nick Craswell
Comments (0)