Sciweavers

COOPIS
2004
IEEE

Minimizing the Network Distance in Distributed Web Crawling

13 years 7 months ago
Minimizing the Network Distance in Distributed Web Crawling
Abstract. Distributed crawling has shown that it can overcome important limitations of the centralized crawling paradigm. However, the distributed nature of current distributed crawlers is currently not fully utilized. The optimal benefits of this approach are usually limited to the sites hosting the crawler. In this work we describe IPMicra, a distributed location aware web crawler that utilizes an IP address hierarchy and allows crawling of links in a near optimal location aware manner. The crawler outperforms earlier distributed crawling approaches without a significant overhead.
Odysseas Papapetrou, George Samaras
Added 20 Aug 2010
Updated 20 Aug 2010
Type Conference
Year 2004
Where COOPIS
Authors Odysseas Papapetrou, George Samaras
Comments (0)