基于混合架构的国产化大数据平台研究与应用  

Research and Application of Localized Big Data Platform Based on Hybrid Architecture

作　　者：王新东 王一大 李昌盛 张亚威 郭炜 Wang Xindong;Wang Yida;Li Changsheng;Zhang Yawei;Guo Wei(China United Network Communications Co.,Ltd.Ji'nan Software Institute,Ji'nan 250100,China)

机构地区：[1]中国联合网络通信有限公司济南软件研究院,济南250100

出　　处：《信息通信技术》2023年第4期75-84,共10页Information and communications Technologies

摘　　要：文章主要研究一种基于混合架构的国产化大数据平台,可以更好地适配国产化ARM主机和麒麟操作系统,解决现有开源Hadoop平台的技术缺陷。通过对现有开源Hadoop平台进行改造并自主研发分布式调度系统和分布式存储系统,使本文大数据平台一方面摆脱底层架构受限问题,可以很好地适配ARM主机和传统X86主机,且全部功能均适配麒麟操作系统,性能稳定可靠;另一方面解决开源Hadoop平台调度系统不完善、交互效率低和单点故障问题。另外通过自主研发统一元数据及安全管理体系,将元数据进行统一规划,使本平台能够节省存储费用,提高安全管控能力。This paper mainly studies a localized big data platform based on a hybrid architecture,which can better adapt to the localized ARM host and Kirin operating system,and solve the technical defects of the existing open source system.By optimizing the open source Hadoop platform and developing a set of distributed scheduling system and distributed storage system,it can not only solve the problem of limited underlying architecture,but also can well adapt to ARM host and traditional X86 host,and all functions are adapted to Kirin operating system,with stable and reliable performance.On the other hand,it solves the problem of imperfect scheduling system,low interaction efficiency and single point of failure of open source Hadoop platform.The self-owned unified metadata and security management system proposed in this paper unifies metadata planning,saves storage costs and improves security management and control capabilities.

关 键 词：大数据平台 国产化 分布式 

分 类 号：TP311.13[自动化与计算机技术—计算机软件与理论]