基于共享的MapReduce多查询优化技术  

Sharing-based multi-query optimization approach using MapReduce

作　　者：赵保学[1] 李战怀[1] 陈群[1] 潘巍[1] 姜涛[1] 金健[1] 

机构地区：[1]西北工业大学计算机学院,西安710072

出　　处：《计算机应用研究》2013年第5期1405-1409,共5页Application Research of Computers

基　　金：国家自然科学基金资助项目(61033007);国家"973"重点基础发展规划基金资助项目(2012CB316203);国家"863"高技术研究发展基金资助项目(2012AA011004);西北工业大学研究生创业种子基金资助项目(Z2012128)

摘　　要：为解决MapReduce处理多个查询时效率低下的问题,提出了一种基于查询共享的MapReduce查询优化方法——ShareOpt优化。通过分析所有查询的操作模式,找出其中共享的子查询部分,并根据子查询的执行顺序构造执行计划有向图(DAG),最终确定一组查询的整体执行计划。通过与Hive和Pig的对比,验证了该方法能够在保证准确性的情况下有效地减少执行步数,提高查询执行的效率。To improve the multi-query processing efficiency for MapReduce, this paper proposed a multi-query optimization approach based on sub-query sharing and merging. Firstly, it analyzed the patterns of all the queries and identified those sub- queries which had sharing opportunities and could be merged. Next it constructed a directed aeyclic graph （DAG） according to the sub-queries execution sequences. It added the non-sharing sub-queries to the DAG graph and finally got the overall exe- cution plan for the queries. Experiment results show that the approach can effectively eliminate unnecessary re-computation and save query execution time.

关 键 词：查询优化 MAPREDUCE 查询共享 执行计划 

分 类 号：TP311.13[自动化与计算机技术—计算机软件与理论]