WWW
2008
ACM
16 years 3 months ago
2008
ACM
A website can regulate search engine crawler access to its content using the robots exclusion protocol, specified in its robots.txt file. The rules in the protocol enable the site...