大数据技术专题讲座(二) 第3讲 面向大数据处理的MapReduce优化技术  

MapReduce Optimization Technologies on Big Data Dealing

作　　者：刘青 鲍爱华[2] 倪桂强[2] 

机构地区：[1]解放军理工大学指挥信息系统学院研究生2队,江苏南京210007 [2]解放军理工大学指挥信息系统学院

出　　处：《军事通信技术》2015年第2期81-87,共7页Journal of Military Communications Technology

基　　金：国家自然科学基金资助项目(61402518)

摘　　要：随着大数据时代的到来,大规模数据的处理技术研究层出不穷。MapReduce计算框架的出现,暂时缓解了大数据的处理难题。虽然MapReduce能够实现大规模数据(通常为PB级甚至EB级)的高效并行处理,但在企业应用环境中,MapReduce底层架构和处理模式上的缺陷逐渐暴露出来,造成了处理效率、执行性能上的一些瓶颈。针对这些不足,结合企业实际处理需求,产生了针对MapReduce的各个方面的优化研究。文中首先阐述了MapReduce的编程模型和实现机制,然后总结了对MapReduce作业调度、实时处理、架构和服务等方面的优化研究,最后对MapReduce各种优化技术进行了总结和展望。With the arrival of the era large-scale data have been emerging in an of big data, researches on the processing technology of endless stream. The emergence of MapReduce tempora- rily alleviates the problem of large data processing. Although MapReduce can realize large-scale data （usually PB or even EB） efficient parallel processing, the drawback on its underlying archi- tecture and processing model has appeared in product environment and caused the bottleneck on processing efficiency and performance. In view of these shortcomings, the various aspects optimi- zation of MapReduce were proposed accordingly. In this paper, the MapReduce programming model and implementation mechanism were firstly described, and then the optimization on sched- uling, real-time processing, architecture and service etc was concluded. Finally, various optimi- zation techniques for MapReduce were summarized and prospected.

关 键 词：大数据 并行编程范式 优化 

分 类 号：TP316.4[自动化与计算机技术—计算机软件与理论]