一种混合计算环境下的MapReduce并行模型  

A novel MapReduce parallel model in hybrid computing environment

作　　者：唐兵[1] 贺海武 

机构地区：[1]湖南科技大学计算机科学与工程学院,湖南湘潭411201 [2]法国里昂高等师范学院并行计算实验室,法国里昂69364

出　　处：《计算机工程与科学》2013年第11期87-93,共7页Computer Engineering & Science

基　　金：湖南省教育厅资助项目(12C0121);湖南科技大学博士科研启动基金(E51097);法国国家科研署项目MapReduce(ANR-10-SEGI-001-01)

摘　　要：提出了一种混合计算环境下的MapReduce并行计算模型,利用该模型可以将高性能集群节点与Internet或Intranet下异构的桌面PC组成混合计算环境,在该混合环境下运行MapReduce任务进行海量数据分析处理,充分利用了大规模桌面PC的计算与存储能力。与Hadoop类似,该模型分为存储层和任务层两层。对该模型及其核心的HybridDFS分布式文件系统和MapReduce算法进行了简单描述,进而设计并实现了一个原型系统,并对其进行了性能测试。测试结果表明,提出的混合计算模型不仅能够实现可靠的MapReduce计算,而且降低了计算的成本开销,具有非常大的潜力。A novel MapReduce computation model in hybrid computing environment is proposed. U- sing this model, high performance cluster nodes and heterogeneous desktop PCs in Internet or Intranet can be integrated to form a hybrid computing environment, where MapReduce tasks can be executed to process large-scale datasets. In this way, the computation and storage capability of large-scale desktop PCs are fully utilized. Similar to the design of Hadoop, this model composes of storage layer and task layer. The paper introduces the architecture of the model briefly and describes the core HybridDFS and the MapReduce algorithms. Then, a prototype system is designed and implemented, and performance e- valuations are accomplished. Evaluation results show that the proposed hybrid computation model is not only able to achieve reliable MapReduce computation, but also reduces the computation cost, hence be- ing a potential effective computation model.

关 键 词：混合计算环境 MAPREDUCE 志愿计算 容错 分布式文件系统 

分 类 号：TP302[自动化与计算机技术—计算机系统结构]