Web search solved?: all result rankings the same?

The objective of this work is to derive quantitative statements about what fraction of web search queries issued to the state-of-the-art commercial search engines lead to excellent results or, on the contrary, poor results. To be able to make such statements in an automated way, we propose a new measure that is based on lower and upper bound analysis over the standard relevance measures. Moreover, we extend this measure to carry out comparisons between competing search engines by introducing the concept of disruptive sets, which we use to estimate the degree to which a search engine solves queries that are not solved by its competitors. We report empirical results on a large editorial evaluation of the three largest search engines in the US market. Categories and Subject Descriptors H.3.3 [Information Storage Systems]: Information Retrieval Systems General Terms Human Factors, Measurement Keywords Web search evaluation, editorial judgments, relevance

• Type: Journal
• Year: 2010
• Where: CIKM
• Authors: Hugo Zaragoza, Berkant Barla Cambazoglu, Ricardo A. Baeza-Yates
• Keywords: CIKM 2010, Commercial Search Engines, Information Technology, Largest Search Engines, Search Engines