Sciweavers

ECIR
2011
Springer

A Methodology for Evaluating Aggregated Search Results

12 years 7 months ago
A Methodology for Evaluating Aggregated Search Results
Aggregated search is the task of incorporating results from different specialized search services, or verticals, into Web search results. While most prior work focuses on deciding which verticals to present, the task of deciding where in the Web results to embed the vertical results has received less attention. We propose a methodology for evaluating an aggregated set of results. Our method elicits a relatively small number of human judgements for a given query and then uses these to facilitate a metric-based evaluation of any possible presentation for the query. An extensive user study with 13 verticals confirms that, when users prefer one presentation of results over another, our metric agrees with the stated preference. By using Amazon’s Mechanical Turk, we show that reliable assessments can be obtained quickly and inexpensively.
Jaime Arguello, Fernando Diaz, Jamie Callan, Ben C
Added 27 Aug 2011
Updated 27 Aug 2011
Type Journal
Year 2011
Where ECIR
Authors Jaime Arguello, Fernando Diaz, Jamie Callan, Ben Carterette
Comments (0)