Sciweavers

WEBI
2007
Springer
13 years 10 months ago
Determining Bias to Search Engines from Robots.txt
Search engines largely rely on robots (i.e., crawlers or spiders) to collect information from the Web. Such crawling activities can be regulated from the server side by deploying ...
Yang Sun, Ziming Zhuang, Isaac G. Councill, C. Lee...