基于Hadoop的高性能海量数据处理平台研究  

High Performance Massive Data Computing Framework Based on Hadoop Cluster

作　　者：翟岩龙[1] 罗壮[1] 杨凯[2] 徐晟晨[1] 

机构地区：[1]北京理工大学计算机学院,北京100081 [2]北京仿真中心,北京100854

出　　处：《计算机科学》2013年第3期100-103,共4页Computer Science

摘　　要：海量数据高性能计算蕴藏着巨大的应用价值,但是目前云计算体系只具有海量数据处理能力,而不具有足够的高性能计算能力。将具有超强并行计算能力的GPU与云计算相融合,提出了基于CPU/GPU协同的异构高性能云计算体系结构。以开源Hadoop为基础,采用注释码的形式对MapReduce函数中需要并行的部分进行标记。通过定制GPU类加载器,将被标记代码转换为CUDA代码并动态编译运行。该平台将GPU的计算能力融合到MapRe-duce框架中,可高效处理海量数据。HPC of massive data presents tremendous value. However, cloud systems still lack HPC computing power. This study improved the HPC ability of cloud computing technology by adding GPU to the cloud system. The proposed platform is based on Hadoop MapReduce programming model, and it defines some OpenMP like directives to annotate MapReduce program. The annotated code will try to be executed in parallel. A GPUClassloader was designed to convert annotated java code regions to CUDA code. With JNI, generated CUDA code and run on the GPUs. The computing re- suits of GPUs can be transferred back to the map function, in the end, the map function finishes the rest computing. The platform can support the user to complete CPU, GPU collaborative large-scale data parallel processing programming conveniently.

关 键 词：CPU GPU协同计算 HADOOP 海量数据处理 高性能计算 

分 类 号：TP309[自动化与计算机技术—计算机系统结构]