Sciweavers

Share
ICDE
2003
IEEE

Flux: An Adaptive Partitioning Operator for Continuous Query Systems

10 years 1 months ago
Flux: An Adaptive Partitioning Operator for Continuous Query Systems
The long-running nature of continuous queries poses new scalability challenges for dataflow processing. CQ systems execute pipelined dataflows that may be shared across multiple queries. The scalability of these dataflows is limited by their constituent, stateful operators ? e.g. windowed joins or grouping operators. To scale such operators, a natural solution is to partition them across a shared-nothing platform. But in the CQ context, traditional, static techniques for partitioned parallelism can exhibit detrimental imbalances as workload and runtime conditions evolve. Longrunning CQ dataflows must continue to function robustly in the face of these imbalances. To address this challenge, we introduce a dataflow operator called Flux that encapsulates adaptive state partitioning and dataflow routing. Flux is placed between producerconsumer stages in a dataflow pipeline to repartition stateful operators while the pipeline is still executing. We present the Flux architecture, along with ...
Mehul A. Shah, Joseph M. Hellerstein, Sirish Chand
Added 01 Nov 2009
Updated 01 Nov 2009
Type Conference
Year 2003
Where ICDE
Authors Mehul A. Shah, Joseph M. Hellerstein, Sirish Chandrasekaran, Michael J. Franklin
Comments (0)
books