Sciweavers

Share
SOSP
2009
ACM

Quincy: fair scheduling for distributed computing clusters

9 years 7 months ago
Quincy: fair scheduling for distributed computing clusters
This paper addresses the problem of scheduling concurrent jobs on clusters where application data is stored on the computing nodes. This setting, in which scheduling computations close to their data is crucial for performance, is increasingly common and arises in systems such as MapReduce, Hadoop, and Dryad as well as many grid-computing environments. We argue that data intensive computation benefits from a fine-grain resource sharing model that differs from the coarser semi-static resource allocations implemented by most existing cluster computing architectures. The problem of scheduling with locality and fairness constraints has not previously been extensively studied under this model of resourcesharing. We introduce a powerful and flexible new framework for scheduling concurrent distributed jobs with fine-grain resource sharing. The scheduling problem is mapped to a graph datastructure, where edge weights and capacities encode the competing demands of data locality, fairness, a...
Michael Isard, Vijayan Prabhakaran, Jon Currey, Ud
Added 17 Mar 2010
Updated 17 Mar 2010
Type Conference
Year 2009
Where SOSP
Authors Michael Isard, Vijayan Prabhakaran, Jon Currey, Udi Wieder, Kunal Talwar, Andrew Goldberg
Comments (0)
books