一种轻量化高可靠分布式数据处理架构  

A Lightweight and Highly Reliable Distributed Data Processing Archit

作　　者：曹芳芳 李兰兰[1] 邹茜薇 崔宇[1] Cao Fangfang;Li Lanlan;Zou Qianwei;Cui Yu(Beijing Aerospace Automatic Control Institute,Beijing 100854,Chin)

机构地区：[1]北京航天自动控制研究所,北京100854

出　　处：《航天控制》2023年第6期50-56,共7页Aerospace Control

摘　　要：针对航天型号体系化高带宽智能总线数据的实时处理需求和现有架构数据整理耗时、运维复杂等问题,开展轻量化的高可靠分布式处理架构研究。通过构建元数据模型库和数据解析器完成数据的清洗和转换;采用基于ORC文件格式的分布式文件系统替代原有Hbase分布式列存储方案,增强平台的易用性和可靠性;基于Spark内存计算模型提升平台的计算、分析能力,结合数据流分片存储机制和索引库实现数据查询的优化加速。该分布式处理架构已应用到某装备系统中,保证了系统高密度试验数据的快速处理、稳定存储和高可靠应用,为高带宽实时数据的快速处理和管理提供了一种解决方案。Regarding the real⁃time processing requirement of the aerospace model systematic high⁃band⁃width intelligent bus data and the time⁃consuming and complex operation and maintenance matters of the existing architecture data processing,a lightweight and highly reliable distributed processing architecture is studied.In order to clean and transform data,metadata model libraries and data parser are built to replace the original Hbase distributed column storage scheme with distributed data store based on ORC file format and to enhance the usability and reliability of the platform;Aiming at enhancing the computing and analy⁃sis capabilities of the platform based on the Spark memory computing model,the data stream fragmentation storage mechanism and the index library are combined to optimize and accelerate the data query.The dis⁃tributed processing architecture has been applied to certain equipment system,which ensures the fast process⁃ing,stable storage and high reliable application of the system high density test data and provides a solution for high bandwidth real⁃time data processing and management.

关 键 词：分布式文件系统 大数据 航天控制 实时处理 

分 类 号：TP319[自动化与计算机技术—计算机软件与理论]