Sciweavers

SIGMETRICS
2000
ACM

Crawler-Friendly Web Servers

13 years 3 months ago
Crawler-Friendly Web Servers
In this paper we study how to make web servers (e.g., Apache) more crawler friendly. Current web servers offer the same interface to crawlers and regular web surfers, even though crawlers and surfers have very different performance requirements. We evaluate simple and easy-to-incorporate modifications to web servers so that there are significant bandwidth savings. Specifically, we propose that web servers export meta-data archives decribing their content.
Onn Brandman, Junghoo Cho, Hector Garcia-Molina, N
Added 19 Dec 2010
Updated 19 Dec 2010
Type Journal
Year 2000
Where SIGMETRICS
Authors Onn Brandman, Junghoo Cho, Hector Garcia-Molina, Narayanan Shivakumar
Comments (0)