Efficient web change monitoring with page digest

The Internet and the World Wide Web have enabled a publishing explosion of useful online information, which has produced the unfortunate side effect of information overload: it is increasingly difficult for individuals to keep abreast of fresh information. In this paper we describe an approach for building a system for efficiently monitoring changes to Web documents. This paper has three main contributions. First, we present a coherent framework that captures different characteristics of Web documents. The system uses the Page Digest encoding to provide a comprehensive monitoring system for content, structure, and other interesting properties of Web documents. Second, the Page Digest encoding enables improved performance for individual page monitors through mechanisms such as short-circuit evaluation, linear time algorithms for document and structure similarity, and data size reduction. Finally, we develop a collection of sentinel grouping techniques based on the Page Digest encoding ...

• Type: Conference
• Year: 2004
• Where: WWW
• Authors: David Buttler, Daniel Rocco, Ling Liu
• Keywords: Information Monitoring Systems, Internet Technology, Large-scale Monitoring Systems, Similar Monitoring Requests, WWW 2004