Sciweavers

WEBI
2007
Springer

Determining Bias to Search Engines from Robots.txt

13 years 9 months ago
Determining Bias to Search Engines from Robots.txt
Search engines largely rely on robots (i.e., crawlers or spiders) to collect information from the Web. Such crawling activities can be regulated from the server side by deploying the Robots Exclusion Protocol in a file called robots.txt. Ethical robots will follow the rules specified in robots.txt. Websites can explicitly specify an access preference for each robot by name. Such biases may lead to a “rich get richer” situation, in which a few popular search engines ultimately dominate the Web because they have preferred access to resources that are inaccessible to others. This issue is seldom addressed, although the robots.txt convention has become a de facto standard for robot regulation and search engines have become an indispensable tool for information access. We propose a metric to evaluate the degree of bias to which specific robots are subjected. We have investigated 7,593 websites covering education, government, news, and business domains, and collected 2,925 distinct r...
Yang Sun, Ziming Zhuang, Isaac G. Councill, C. Lee
Added 09 Jun 2010
Updated 09 Jun 2010
Type Conference
Year 2007
Where WEBI
Authors Yang Sun, Ziming Zhuang, Isaac G. Councill, C. Lee Giles
Comments (0)