基于Hadoop技术的加速器大数据安全存储与高效分析系统设计  

Design secure storage and efficient analysis system for accelerator big data based on Hadoop technology

作　　者：赵子晨 杨锋[1] 郭玉辉[1] 陈又新[1] 李钊扬 刘海涛[1] ZHAO Zichen;YANG Feng;GUO Yuhui;CHEN Youxin;LI Zhaoyang;LIU Haitao(Institute of Modern Physics,Chinese Academy of Sciences,Lanzhou 730000,China;School of Computer Science and Technology,University of Chinese Academy of Sciences,Beijing 100049,China)

机构地区：[1]中国科学院近代物理研究所,甘肃兰州730000 [2]中国科学院大学计算机科学与技术学院,北京100049

出　　处：《现代电子技术》2024年第8期9-17,共9页Modern Electronics Technique

基　　金：中国科学院战略性先导科技专项(B类)(XDB34010000)。

摘　　要：为了解决当前加速器控制系统在数据管理方面所面临的海量数据安全存储和高效分析处理的问题,在现有的基础上引入了Hadoop大数据框架,并结合大数据其他相关组件,构建一个分布式数据仓库系统。文章详细阐述了数据仓库的搭建过程,包括软硬件架构以及将数据从现有数据库抽取、转换和加载到数据仓库的方案。特别是针对系统归档数据的存储和分析需求,根据实际应用场景设计一个基于HBase的存储解决方案。在系统部署完成后,进行了吞吐量测试,并与当前使用的传统数据库进行性能对比。测试结果显示,基于Hadoop的数据仓库系统在海量数据存储、高性能查询以及数据分析处理方面都表现出明显的优势。这一改进为加速器控制系统提供了更强大的数据管理和处理能力,有望为加速器在未来的发展提供更多可能性。In order to address the challenges of massive data secure storage and efficient analysis in the data management of the current accelerator control system,the Hadoop big data framework is introduced on the existing foundation.In combination of various big data components,a distributed data warehouse system is constructed.The process of building a data warehouse is elaborated,including software and hardware architecture,as well as a solution for extracting,converting,and loading data from existing databases into the data warehouse.An HBase based storage solution is designed based on practical application scenarios for the storage and analysis requirements of system archived data.After the system deployment,throughput testing was conducted and performance comparisons were made with the traditional databases currently in use.The testing results show that the the Hadoop based data warehouse system has shown significant advantages in massive data storage,high-performance queries,and data analysis and processing.This improvement provides stronger data management and processing capabilities for accelerator control systems,and is expected to provide more possibilities for the future development of accelerator control.

关 键 词：数据仓库 Hadoop技术 加速器 大数据 安全存储 HBASE ETL 

分 类 号：TN911-34[电子电信—通信与信息系统] TP311.1[电子电信—信息与通信工程] TL53[自动化与计算机技术—计算机软件与理论]