Modeling skew in data streams

14 years 4 months ago

Download www.research.rutgers.edu

Data stream applications have made use of statistical summaries to reason about the data using nonparametric tools such as histograms, heavy hitters, and join sizes. However, relatively little attention has been paid to modeling stream data parametrically, despite the potential this approach has for mining the data. The challenges to do model fitting at streaming speeds are both technical ? how to continually find fast and reliable parameter estimates on high speed streams of skewed data using small space ? and conceptual ? how to validate the goodness-of-fit and stability of the model online. In this paper, we show how to fit hierarchical (binomial multifractal) and non-hierarchical (Pareto) power-law models on a data stream. We address the technical challenges using an approach that maintains a sketch of the data stream and fits least-squares straight lines; it yields algorithms that are fast, space-efficient, and provide approximations of parameter value estimates with a priori qua...

Flip Korn, S. Muthukrishnan, Yihua Wu

Real-time Traffic