Sciweavers

JCST
2010

Queue Waiting Time Aware Dynamic Workflow Scheduling in Multicluster Environments

12 years 10 months ago
Queue Waiting Time Aware Dynamic Workflow Scheduling in Multicluster Environments
Abstract Workflows are prevailing in scientific computation. Multicluster environments emerge and provide more resources, benefiting workflows but also challenging the traditional workflow scheduling heuristics. In a multicluster environment, each cluster has its own independent workload management system. Jobs are queued up before getting executed, they experience different resource availability and wait time if dispatched to different clusters. However, existing scheduling heuristics neither consider the queue wait time nor balance the performance gain with data movement cost. The proposed algorithm leverages the advancement of queue wait time prediction techniques and empirically studies if the tunability of resource requirements helps scheduling. The extensive experiment with both real workload traces and test bench shows that the queue wait time aware algorithm improves workflow performance by 3 to 10 times in terms of average makespan with relatively very low cost of data movemen...
Zhi-Feng Yu, Wei-Song Shi
Added 19 May 2011
Updated 19 May 2011
Type Journal
Year 2010
Where JCST
Authors Zhi-Feng Yu, Wei-Song Shi
Comments (0)