Comparing click-through data to purchase decisions for retrieval evaluation

Traditional retrieval evaluation uses explicit relevance judgments which are expensive to collect. Relevance assessments inferred from implicit feedback such as click-through data can be collected inexpensively, but may be less reliable. We compare assessments derived from click-through data to another source of implicit feedback that we assume to be highly indicative of relevance: purchase decisions. Evaluating retrieval runs based on a log of an audiovisual archive, we find agreement between system rankings and purchase decisions to be surprisingly high. Categories and Subject Descriptors: H.3 [Information Storage and Retrieval]: H.3.3 Information Search and Retrieval General Terms: Experimentation, Human Factors

• Type: Conference
• Year: 2010
• Where: SIGIR
• Authors: Katja Hofmann, Bouke Huurnink, Marc Bron, Maarten de Rijke
• Keywords: Click-through Data, Explicit Relevance Judgments, Implicit Feedback, Information Management, SIGIR 2010