一种基于SSD的高性能Hadoop系统的设计与应用  

Design and Application of a High Performance SSD-based Hadoop System

作　　者：陈丽[1] 王锐[2] 胡刚 

机构地区：[1]广东交通职业技术学院,广东广州510650 [2]中国移动通信集团广东有限公司,广东广州510623 [3]星环信息科技(上海)有限公司,上海200233

出　　处：《广东水利电力职业技术学院学报》2016年第1期39-44,共6页Journal of Guangdong Polytechnic of Water Resources and Electric Engineering

基　　金：广东省高职高专云计算与大数据专业委员会教育科研课题(GDYJSKT14-03)

摘　　要：在大数据平台中,随第一代Hadoop出现的基于磁盘的批处理计算框架Map Reduce的局限日益显现:数据读入和中间结果数据均依赖于大量磁盘I/O,性能有限。基于内存计算的Spark成为当前重点,对比Map Reduce性能可提升10-100倍,但内存使用的成本依旧高昂。采用超高速且相对廉价的SSD作为大数据的缓存解决方案,阐述了SSD和HDD的混合存储架构;同时,结合高效的Spark计算引擎和行列混合式文件结构等优化技术,设计了一种基于SSD的高性能Hadoop系统,能有效地解决大数据计算系统的性能问题和存储空间问题,实验结果显示取得数倍性能提升。In the big data platform, the deficiency of MapReduce begins to emerge, which is designed for batch computation with the first generation of Ha-doop. Data read and intermediate results depend a lot on disk I/0, and the performance is limited. The memory based computation, spark, is more important. While the cost is too high, though it is 10-100 times faster than Hadoop MapReduee. This papers proposes a storage solution with SSD as big data caches, which is faster and cheaper. A SSD/HDD hybrid storage architecture is explored. Meanwhile, with the optimization techniques of efficient Spark computation engine and hybrid row-column data placement structure respectively, this paper proposes a high performance Hadoop system based on SSD, an effective solution to processing performance and storage space in the big data computing system. The experimental results show that the improvement the system is significant.

关 键 词：大数据 HADOOP SPARK SSD 行列混合式存储 

分 类 号：TP311[自动化与计算机技术—计算机软件与理论]