Data locality in MapReduce: A network perspective

2 years 9 months ago
Data locality in MapReduce: A network perspective
Data locality, a critical consideration for the performance of task scheduling in MapReduce, has been addressed in the literature by increasing the number of locally processed tasks. In this paper, we view the data locality problem from a network perspective. The key observation is that if we make appropriate use of the network to route the data chunk to the machine where it will be processed in advance, then processing a remote task is the same as processing a local task. However, to benefit from such a strategy, we must (i) balance the tasks assigned to local machines and those assigned to remote machines, and (ii) design the routing algorithm to avoid network congestion. Taking these challenges into consideration, we propose a scheduling/routing algorithm, named the Joint Scheduler, which utilizes both the computing resources and the communication network efficiently. We prove that the Joint Scheduler is throughput optimal; i.e., it supports any load that is supportable by any oth...
Weina Wang, Lei Ying
Added 08 Apr 2016
Updated 08 Apr 2016
Type Journal
Year 2016
Where PE
Authors Weina Wang, Lei Ying
Comments (0)