Automatic ranking of information retrieval systems using data fusion

12 years 3 months ago
Automatic ranking of information retrieval systems using data fusion
Measuring effectiveness of information retrieval (IR) systems is essential for research and development and for monitoring search quality in dynamic environments. In this study, we employ new methods for automatic ranking of retrieval systems. In these methods, we merge the retrieval results of multiple systems using various data fusion algorithms, use the top-ranked documents in the merged result as the ``(pseudo) relevant documents,'' and employ these documents to evaluate and rank the systems. Experiments using Text REtrieval Conference (TREC) data provide statistically significant strong correlations with human-based assessments of the same systems. We hypothesize that the selection of systems that would return documents different from the majority could eliminate the ordinary systems from data fusion and provide better discrimination among the documents and systems. This could improve the effectiveness of automatic ranking. Based on this intuition, we introduce a new me...
Rabia Nuray, Fazli Can
Added 13 Dec 2010
Updated 13 Dec 2010
Type Journal
Year 2006
Where IPM
Authors Rabia Nuray, Fazli Can
Comments (0)