Sciweavers

SIGMOD
2010
ACM

Online aggregation and continuous query support in MapReduce

13 years 8 months ago
Online aggregation and continuous query support in MapReduce
MapReduce is a popular framework for data-intensive distributed computing of batch jobs. To simplify fault tolerance, the output of each MapReduce task and job is materialized to disk before it is consumed. In this demonstration, we describe a modified MapReduce architecture that allows data to be pipelined between operators. This extends the MapReduce programming model beyond batch processing, and can reduce completion times and improve system utilization for batch jobs as well. We demonstrate a modified version of the Hadoop MapReduce framework that supports online aggregation, which allows users to see “early returns” from a job as it is being computed. Our Hadoop Online Prototype (HOP) also supports continuous queries, which enable MapReduce programs to be written for applications such as event monitoring and stream processing. HOP retains the fault tolerance properties of Hadoop, and can run unmodified user-defined MapReduce programs. Categories and Subject Descriptors H....
Tyson Condie, Neil Conway, Peter Alvaro, Joseph M.
Added 18 Jul 2010
Updated 18 Jul 2010
Type Conference
Year 2010
Where SIGMOD
Authors Tyson Condie, Neil Conway, Peter Alvaro, Joseph M. Hellerstein, John Gerth, Justin Talbot, Khaled Elmeleegy, Russell Sears
Comments (0)