Hadoop海量数据迁移系统开发及应用  

Development and Application of Hadoop Massive Data Migration System

作　　者：尹乔 魏占辰 黄秋兰[1] 孙功星[1] 石京燕[1] YIN Qiao;WEI Zhanchen;HUANG Qiulan;SUN Gongxing;SHI Jingyan(Institute of High Energy Physics,Chinese Academy of Sciences,Beijing 100049,China;University of Chinese Academy of Sciences,Beijing 100049,China)

机构地区：[1]中国科学院高能物理研究所,北京100049 [2]中国科学院大学,北京100049

出　　处：《计算机工程与应用》2019年第13期66-71,共6页Computer Engineering and Applications

基　　金：国家自然科学基金(No.11775249,No.11775250)

摘　　要：当前高能物理实验产生的数据量越来越大,利用大数据处理平台Hadoop进行高能物理数据处理时,面临数据迁移的实际需求,而现有迁移工具不支持HDFS与其他文件系统间的数据传输,性能存在明显缺陷。从高能物理数据同步、归档等需求出发,设计和实现了一个通用的海量数据迁移系统,通过扩展HDFS数据访问方式,使用Map-Reduce直接在HDFS数据节点和其他存储系统/介质之间迁移数据。此外,系统设计实现了动态优先级调度模型,进行多任务的动态优先级评定和选取。该系统已经应用于大型高海拔空气簇射观测站(LHAASO)宇宙线等物理实验中的数据迁移,实际运行结果表明系统性能良好,能够满足各个实验的数据迁移需求。With more and more data generated by High Energy Physics(HEP)experiments, Hadoop has been a solution for HEP data analysis while facing with the demand of data migration. However, existing data migration tools do not support data transmission between HDFS and other file systems, and have obvious performance deficiency. Based on the requirements of high-energy physical data synchronization and archiving, this paper designs and implements a universal mass data migration system, which uses MapReduce to directly move data between HDFS and other storage systems or media by extending the HDFS data access methods. In addition, dynamic priority scheduling model is proposed to do multi-tasks dynamic priority assignment and selection. The system has been applied to the data migration in LHAASO experiment, and the actual operation results indicate that the system achieves good performance and meets the data migration requirements of various experiments.

关 键 词：高能物理 数据迁移 GridFTP协议 动态优先级调度 多属性决策 Hadoop系统 

分 类 号：TP311[自动化与计算机技术—计算机软件与理论]