一种具有容错机制的MapReduce模型研究与实现  

A MapReduce System with Fault-Tolerant Mechanism

作　　者：史椸[1] 耿晨[2] 齐勇[1] 

机构地区：[1]西安交通大学电子与信息工程学院,西安710049 [2]中航工业西安飞行自动控制研究所,西安710065

出　　处：《西安交通大学学报》2014年第2期1-7,共7页Journal of Xi'an Jiaotong University

基　　金：国家自然科学基金资助项目(60933003);教育部高等学校博士学科点专项科研基金资助项目(20120201110010)

摘　　要：针对传统MapReduce模型的容错机制对错误的处理效率低下等问题,提出了一种基于多核虚拟机的具有容错机制的MapReduce模型。该模型使用检查点机制进行错误恢复,并只对中间结果和必要的状态信息进行保存;利用虚拟机在隔离内存中保存中间结果;根据用户的需要及系统的负载情况动态调整系统中工作节点的个数。通过在SUN的32核、主频为2.38GHz、内存为128GB服务器上的测试,结果表明:与传统MapReduce模型相比,改进MapReduce模型降低了通信上的开销,提高了MapReduce运行过程的可靠性和错误恢复的性能,虚拟机监控器可以完全控制和管理多核平台的内存,使操作系统无法直接访问隔离的内存,数据恢复不会受到操作系统内部各种错误的影响,保证了恢复数据的安全性。A MapReduce with fault-tolerant mechanism based on multi-core virtual machine is proposed to solve the problem that the simple fault-tolerant mechanism in traditional MapReduce is prone for duplicate calculation in error processing. The system uses checkpoint mechanism to perform error recovery and only saves intermediate results and necessary state information. It stores the intermediate results in isolated memory through virtual machine. The number of worker nodes is dynamically adjusted according to the user＇s need and the system load. Experiments are conducted on 32-core, 2.38 GHz, 128 GB SUN server and the results show that the proposed MapReduce reduces communication cost, and improves system reliability and error recovery performance. Virtual Machine Monitor can entirely control and manage the memory of a multi-core system and disable OS to visit the isolated memory directly. The security of data recovery is guaranteed by avoiding affection of errors occurring inside the OS.

关 键 词：多核 虚拟机 容错 MAPREDUCE 

分 类 号：TP311[自动化与计算机技术—计算机软件与理论]