Finding global icebergs over distributed data sets

Finding icebergs ? items whose frequency of occurrence is above a certain threshold ? is an important problem with a wide range of applications. Most of the existing work focuses on iceberg queries at a single node. However, in many real-life applications, data sets are distributed across a large number of nodes. Two na?ive approaches might be considered. In the first, each node ships its entire data set to a central server, and the central server uses single-node algorithms to find icebergs. But it may incur prohibitive communication overhead. In the second, each node submits local icebergs, and the central server combines local icebergs to find global icebergs. But it may fail because in many important applications, globally frequent items may not be frequent at any node. In this work, we propose two novel schemes that provide accurate and efficient solutions to this problem: a sampling-based scheme and a counting-sketch-based scheme. In particular, the latter scheme incurs a commun...

• Type: Conference
• Year: 2006
• Where: PODS
• Authors: Qi Zhao, Mitsunori Ogihara, Haixun Wang, Jun Xu
• Keywords: Central Server, Combines Local Icebergs, Database, Global Icebergs, PODS 2006