Adaptive Control of Extreme-scale Stream Processing Systems

Abstract— Distributed stream processing systems offer a highly scalable and dynamically configurable platform for time-critical applications ranging from real-time, exploratory data mining to high performance transaction processing. Resource management for distributed stream processing systems is complicated by a number of factors – processing elements are constrained by their producer-consumer relationships, data and processing rates can be highly bursty, and traditional measures of effectiveness, such as utilization, can be misleading. In this paper, we propose a novel distributed, adaptive control algorithm that maximizes weighted throughput while ensuring stable operation in the face of highly bursty workloads. Our algorithm is designed to meet the challenges of extreme-scale stream processing systems, where overprovisioning is not an option, by making the best use of resources even when the proffered load is greater than available resources. We have implemented our algorithm ...

• Type: Conference
• Year: 2006
• Where: ICDCS
• Authors: Lisa Amini, Navendu Jain, Anshul Sehgal, Jeremy Silber, Olivier Verscheure
• Keywords: Algorithm, Distributed And Parallel Computing, ICDCS 2006, Stream Processing, Stream Processing Systems