Automatic Optimization for MapReduce Programs

10 years 8 months ago
Automatic Optimization for MapReduce Programs
The MapReduce distributed programming framework has become popular, despite evidence that current implementations are inefficient, requiring far more hardware than a traditional relational databases to complete similar tasks. MapReduce jobs are amenable to many traditional database query optimizations (B+Trees for selections, column-storestyle techniques for projections, etc), but existing systems do not apply them, substantially because free-form user code obscures the true data operation being performed. For example, a selection in SQL is easily detected, but a selection in a MapReduce program is embedded in Java code along with lots of other program logic. We could ask the programmer to provide explicit hints about the program’s data semantics, but one of MapReduce’s attractions is precisely that it does not ask the user for such information. This paper covers Manimal, which automatically analyzes MapReduce programs and applies appropriate dataaware optimizations, thereby requi...
Eaman Jahani, Michael J. Cafarella, Christopher R&
Added 13 May 2011
Updated 13 May 2011
Type Journal
Year 2011
Where CORR
Authors Eaman Jahani, Michael J. Cafarella, Christopher Ré
Comments (0)