基于流式计算的空间科学卫星数据实时处理  

Real-time processing of space science satellite data based on stream computing

作　　者：孙小涓[1,2,3] 石涛 胡玉新[1,2,3] 佟继周[3,4] 李冰 宋峣 SUN Xiaojuan;SHI Tao;HU Yuxin;TONG Jizhou;LI Bing;SONG Yao(Institute of Electronics,Chinese Academy of Sciences,Beijing 100190,China;Key Laboratory of Technology in Geo-spatial Information Processing and Application System,Chinese Academy of Sciences,Beijing 100190,China;University of Chinese Academy of Sciences,Beijing 100049,China;National Space Science Center,Chinese Academy of Sciences,Beijing 100190,China)

机构地区：[1]中国科学院电子学研究所,北京100190 [2]中国科学院空间信息处理与应用系统技术重点实验室,北京100190 [3]中国科学院大学,北京100049 [4]中国科学院国家空间科学中心,北京100190

出　　处：《计算机应用》2019年第6期1563-1568,共6页journal of Computer Applications

基　　金：中国科学院十三五信息化专项(XXH13505-04);北京市科技计划项目(Z181100002918002)~~

摘　　要：针对空间科学卫星探测数据的实时处理要求越来越高的问题,提出一种基于流计算框架的空间科学卫星数据实时处理方法。首先,根据空间科学卫星数据处理特点对数据流进行抽象分析;然后,对各处理单元的输入输出数据结构进行重新定义;最后,基于流计算框架Storm设计数据流处理并行结构,以适应大规模数据并行处理和分布式计算的要求。对应用该方法开发的空间科学卫星数据处理系统进行测试分析,测试结果显示,在相同条件下数据处理时间比原有系统缩短了一半;数据局部性策略比轮询策略具有更高的吞吐率,数据元组吞吐率平均提高29%。可见采用流式计算框架能够大幅缩短数据处理延迟,提高空间科学卫星数据处理系统的实时性。Concerning the increasingly high real-time processing requirement of space science satellite observed data, a real-time processing method of space science satellite data based on stream computing framework was proposed. Firstly, the data stream was abstractly analyzed according to the data processing characteristics of space science satellite. Then, the input and output data structures of each processing unit were redefined. Finally, the parallel data stream processing structure was designed based on the stream computing framework Storm to meet the requirements of parallel processing and distributed computing of large-scale data. The developed system for space science satellite data processing applying with this method was tested and analyzed. The results show that the data processing time is half of that of the original system under same conditions and the data localization strategy has higher throughput than round-robin strategy with the data tuple throughput increased by 29% on average. It can be seen that the use of stream computing framework can greatly shorten the data processing delay and improve the real-time performance of the space science satellite data processing system.

关 键 词：流式计算 数据流 STORM 空间科学卫星 数据处理 

分 类 号：TP319[自动化与计算机技术—计算机软件与理论]