Design of a crawler with bounded bandwidth

This paper presents an algorithm to bound the bandwidth of a Web crawler. The crawler collects statistics on the transfer rate of each server to predict the expected bandwidth use for future downloads. The prediction allows us to activate the optimal number of fetcher threads in order to exploit the assigned bandwidth. The experimental results show the effectiveness of the proposed technique. Categories and Subject Descriptors: H.3 [Information Systems]: Information Storage and Retrieval General Terms: Design, Experimentation, Performance

• Type: Conference
• Year: 2004
• Where: WWW
• Authors: Michelangelo Diligenti, Marco Maggini, Filippo Maria Pucci, Franco Scarselli
• Keywords: Crawler Collects Statistics, Internet Technology, Retrieval General Terms, Web Crawler, WWW 2004