基于Spark的有效载荷参数解析处理方法  

Processing method of parameter analysis of payload based on Spark

作　　者：张文彬 王春梅 王静 陈托 智佳 ZHANG Wen-bin;WANG Chun-mei;WANG Jing;CHEN Tuo;ZHI Jia(National Space Science Center,Chinese Academy of Sciences,Beijing 100190,China;School of Computer Science and Technology,University of Chinese Academy of Sciences,Beijing 100049,China)

机构地区：[1]中国科学院国家空间科学中心,北京100190 [2]中国科学院大学计算机科学与技术学院,北京100049

出　　处：《计算机工程与设计》2020年第2期587-591,共5页Computer Engineering and Design

摘　　要：为解决单机多线程有效载荷实时参数解析处理方法存在的吞吐率低、扩展能力弱问题,提出一种基于Spark集群的有效载荷实时参数解析处理方法。采用Kafka和Spark相结合的处理方式,利用Kafka将有效载荷实时数据转换为消息队列流,Spark通过Kafka获取消息队列流的数据,利用内存对数据进行迭代运算,提高数据运算速度,实现实时的、高吞吐率的有效载荷参数解析。仿真结果表明,使用该方法在实时吞吐率上较单机多线程处理方法有较高提升,能有效满足实时参数解析的要求。To solve low throughput and weak expansion capability of the processing method of single-machine multi-thread payload real-time parameter analysis,a processing method of real-time parameter analysis of payload based on Spark cluster was presented.A combination of Kafka and Spark was used.Kafka was used to convert real-time payload data into message queue flow.Spark obtained data from a message queue flow through Kafka,the memory was used to iterate over the data to improve the data operation speed and achieve real-time and high throughput parameter analysis.The simulation results show that the proposed method has higher real-time throughput rate and it can effectively meet the requirements of real-time parameter analysis.

关 键 词：有效载荷 参数解析 实时数据处理 吞吐率 集群 

分 类 号：TP311[自动化与计算机技术—计算机软件与理论]