Modeling skew in data streams

12 years 10 months ago
Modeling skew in data streams
Data stream applications have made use of statistical summaries to reason about the data using nonparametric tools such as histograms, heavy hitters, and join sizes. However, relatively little attention has been paid to modeling stream data parametrically, despite the potential this approach has for mining the data. The challenges to do model fitting at streaming speeds are both technical ? how to continually find fast and reliable parameter estimates on high speed streams of skewed data using small space ? and conceptual ? how to validate the goodness-of-fit and stability of the model online. In this paper, we show how to fit hierarchical (binomial multifractal) and non-hierarchical (Pareto) power-law models on a data stream. We address the technical challenges using an approach that maintains a sketch of the data stream and fits least-squares straight lines; it yields algorithms that are fast, space-efficient, and provide approximations of parameter value estimates with a priori qua...
Flip Korn, S. Muthukrishnan, Yihua Wu
Added 08 Dec 2009
Updated 08 Dec 2009
Type Conference
Year 2006
Authors Flip Korn, S. Muthukrishnan, Yihua Wu
Comments (0)