Sciweavers

Share
ICDCS
2011
IEEE

YSmart: Yet Another SQL-to-MapReduce Translator

9 years 2 months ago
YSmart: Yet Another SQL-to-MapReduce Translator
— MapReduce has become an effective approach to big data analytics in large cluster systems, where SQL-like queries play important roles to interface between users and systems. However, based on our Facebook daily operation results, certain types of queries are executed at an unacceptable low speed by Hive (a production SQL-to-MapReduce translator). In this paper, we demonstrate that existing SQL-to-MapReduce translators that operate in a one-operation-to-one-job mode and do not consider query correlations cannot generate high-performance MapReduce programs for certain queries, due to the mismatch between complex SQL structures and simple MapReduce framework. We propose and develop a system called YSmart, a correlation aware SQL-to-MapReduce translator. YSmart applies a set of rules to use the minimal number of MapReduce jobs to execute multiple correlated operations in a complex query. YSmart can significantly reduce redundant computations, I/O operations and network transfers comp...
Rubao Lee, Tian Luo, Yin Huai, Fusheng Wang, Yongq
Added 24 Dec 2011
Updated 24 Dec 2011
Type Journal
Year 2011
Where ICDCS
Authors Rubao Lee, Tian Luo, Yin Huai, Fusheng Wang, Yongqiang He, Xiaodong Zhang
Comments (0)
books