基于Spark Streaming的气象自动站实时流处理与存储系统  

Real-time Stream Processing and Storage System of Automatic Weather Station Data Based on Spark Streaming

作　　者：马彬 李玉涛 许琪 MA Bin;LI Yu-tao;XU Qi(Jiangsu Meteorological Information Center,Nanjing 210005,China;Jiangsu Climate Center,Nanjing 210005,China)

机构地区：[1]江苏省气象信息中心,江苏南京210005 [2]江苏省气侯中心,江苏南京210005

出　　处：《计算机技术与发展》2023年第3期207-214,共8页Computer Technology and Development

基　　金：江苏省气象局面上基金项目(KM202002);中国气象局小型基建项目(QJ2017006)。

摘　　要：在当前大数据技术蓬勃发展的时代,人们对气象数据的实时处理、数据质量、数据存储及大规模查询等要求也越来越高。针对现有气象自动站数据业务落地环节多,任务处理耦合紧但系统部署分散等问题,文中基于Spark Streaming的流式计算框架,研究使用Flume解析收集自动站原始数据,在Spark Streaming中设计融入自动站数据质控算法,最终通过对分布式数据库存储的表设计,使气象自动站数据具备高效率、高质量、高可靠的应用服务能力。性能测试结果表明,基于Spark Streaming的气象自动站数据实时流处理与存储系统,数据从文件采集、解码、流处理至入库的全流程能够在秒级完成,TB级数据查询响应为毫秒级,加权查询为秒级,完全满足自动站数据业务应用需求,从而为进一步提高气象自动站数据质量与服务水平提供基础支撑。In the current era of vigorous development of big data technology, people have higher and higher requirements for real-time processing, data quality, data storage and large-scale query of meteorological data. Aiming at these problems of automatic weather station data with many operation nodes, tightly-coupled task processing, and decentralized system deployment, based on the streaming computing framework of Spark Streaming, the Flume is used to analyze and collect the raw data of automatic weather stations, then the data quality control algorithms are designed and integrated into Spark Streaming. Finally, the table of distributed database storage is designed to make the automatic weather station data with high-efficiency, high-quality, and high-reliability application service capabilities. The test results show that the real-time stream processing and storage system of automatic weather station data based on Spark Streaming can complete the process of data collection, decoding, stream processing and storage in milliseconds, with the query response of TB level data in milliseconds, weighted query in second level, which fully meets the requirements of automatic station data in operation application. Thus it will provide a basic support for further improving the data quality and service of automatic weather stations.

关 键 词：气象自动站数据 Spark Streaming 实时处理 FLUME 分布式数据库 

分 类 号：TP399[自动化与计算机技术—计算机应用技术]