Sciweavers

Share
IEEECLOUD
2013

Using a Tunable Knob for Reducing Makespan of MapReduce Jobs in a Hadoop Cluster

5 years 11 months ago
Using a Tunable Knob for Reducing Makespan of MapReduce Jobs in a Hadoop Cluster
—The MapReduce framework and its open source implementation Hadoop have become the defacto platform for scalable analysis on large data sets in recent years. One of the primary concerns in Hadoop is how to minimize the completion length (i.e., makespan) of a set of MapReduce jobs. The current Hadoop only allows static slot configuration, i.e., fixed numbers of map slots and reduce slots throughout the lifetime of a cluster. However, we found that such a static configuration may lead to low system resource utilizations as well as long completion length. Motivated by this, we propose a simple yet effective scheme which uses slot ratio between map and reduce tasks as a tunable knob for reducing the makespan of a given set. By leveraging the workload information of recently completed jobs, our scheme dynamically allocates resources (or slots) to map and reduce tasks. We implemented the presented scheme in Hadoop V0.20.2 and evaluated it with representative MapReduce benchmarks at Amaz...
Yi Yao, Jiayin Wang, Bo Sheng, Ningfang Mi
Added 28 Apr 2014
Updated 28 Apr 2014
Type Journal
Year 2013
Where IEEECLOUD
Authors Yi Yao, Jiayin Wang, Bo Sheng, Ningfang Mi
Comments (0)
books