Sciweavers

Share
DEBS
2010
ACM

Workload characterization for operator-based distributed stream processing applications

10 years 4 months ago
Workload characterization for operator-based distributed stream processing applications
Operator-based programming languages provide an effective development model for large scale stream processing applications. A stream processing application consists of many runtime deployable software processing elements (PE) that work in flows to process incoming messages. Operators (OP) are logical building blocks hosted by PEs. One or more OPs can be fused into a PE at compile-time. Performance optimization for our streaming system includes compile-time fusion optimization and runtime PE-to-host deployment. One of the goals of an optimized stream application is to use minimal computing resource to sustain maximal message throughput. Characterizing the resource usage of PEs is critical for performance optimization. During compile-time optimization, OP-level resource usage is used to predict the resource usage of fused PEs. When starting an application, PE-level resource usage is used as an initial estimation by the scheduler. In this paper, we propose an efficient workload character...
Xiaolan J. Zhang, Sujay Parekh, Bugra Gedik, Henri
Added 08 Sep 2010
Updated 08 Sep 2010
Type Conference
Year 2010
Where DEBS
Authors Xiaolan J. Zhang, Sujay Parekh, Bugra Gedik, Henrique Andrade, Kun-Lung Wu
Comments (0)
books