Data throttling for data-intensive workflows

11 years 7 months ago
Data throttling for data-intensive workflows
— Existing workflow systems attempt to achieve high performance by intelligently scheduling tasks on resources, sometimes even attempting to move the largest data files on the highest-capacity links. However, such approaches are inherently limited, in that there is only minimal control available regarding the arrival time and rate of data transfer between nodes, resulting in unbalanced workflows in which one task is idle while waiting for data to arrive. This paper describes a data throttling framework that can be exploited by workflow systems to uniquely regulate the rate of data transfers between the workflow tasks via a specially-created QoS-enabled GridFTP server. Our workflow planner constructs a schedule that both specifies when/where individual tasks are to be executed, as well as when and at what rate data is to be transferred. Simulation results involving a simple workflow indicate that our system can achieve a 30% speedup when nodes show a computation/communication ratio of...
Sang-Min Park, Marty Humphrey
Added 31 May 2010
Updated 31 May 2010
Type Conference
Year 2008
Where IPPS
Authors Sang-Min Park, Marty Humphrey
Comments (0)