Distributed location aware web crawling

11 years 6 months ago
Distributed location aware web crawling
Distributed crawling has shown that it can overcome important limitations of the today's crawling paradigm. However, the optimal benefits of this approach are usually limited to the sites hosting the crawler. In this work, we propose a location-aware method, called IPMicra, that utilizes an IP address hierarchy, and allows crawling of links in a near optimal location aware manner. Categories and Subject Descriptors H.3.4 [Systems and Software]: Distributed Systems General Terms Performance Keywords location aware web crawling, distributed web crawling
Odysseas Papapetrou, George Samaras
Added 22 Nov 2009
Updated 22 Nov 2009
Type Conference
Year 2004
Where WWW
Authors Odysseas Papapetrou, George Samaras
Comments (0)