Near-Optimal Compression of Probabilistic Counting Sketches for Networking Applications

Sketches--data structures for probabilistic, duplicate insensitive counting--are central building blocks of a number of recently proposed network protocols, for example in the context of wireless sensor networks. They can be used to perform robust, distributed data aggregation in a broad range of settings and applications. However, the structure of these sketches is very redundant, making effective compression vital if they are to be transmitted over a network. Here, we propose lossless compression schemes for two types of sketches, Flajolet-Martin sketches and HyperLogLog sketches. They use arithmetic coding as a basis. Analysis and simulations show that compression very close to the entropy limit can be achieved, with an algorithm that is simple enough to be employed even on very resource-constrained hardware. The proposed method outperforms existing compression schemes for FlajoletMartin sketches by far. Furthermore, we point out some surprising parallels between compressed Flajole...

• Type: Conference
• Year: 2007
• Where: DIALM
• Authors: Björn Scheuermann, Martin Mauve
• Keywords: Algorithms, Compression Schemes, DIALM 2007, Flajolet-martin Sketches, Hyperloglog Sketches