基于大数据技术的气象业务监视数据采集处理  

Data Collection and Processing Framework for Operation Monitoring Based on Big Data Technology

作　　者：曾乐[1] 孙超[1] 张来恩[1] 陈文琴[1] ZENG Le;SUN Chao;ZHANG Lai-en;CHEN Wen-qin(National Meteorological Information Center,Beijing100081,China)

机构地区：[1]国家气象信息中心,北京100081

出　　处：《计算机仿真》2021年第7期181-188,共8页Computer Simulation

摘　　要：气象数据生产过程中秒级数据流量达到6万次/秒,为了对海量气象数据进行实时监控,快速定位数据观测、传输、处理、服务全流程中各环节故障,研发了对监视数据的采集和处理框架。基于REST接口和Flume框架实时采集原始监视信息,采用Kafka实现监视数据流的缓冲和持久化存储,在Spark Streaming流式计算平台上实现对监视数据的预处理、指标计算,并对告警事件进行归并、压缩等处理,最终生成面向运维人员的告警。同时、上述系统采用故障仿真压测技术,对系统可能出现的故障进行了模拟压力测试。实验结果表明,上述框架能有效地解决海量监视数据的高效采集和处理,能够实时捕捉故障并进行有效分析与排除,其处理时效和准确性满足气象综合业务实时监控的需求。In the process of meteorological data production,the second level data flow reaches 60,000 times per second.In order to monitor the large amount of meteorological data in real time and locate the faults in the whole process of data,from observation,transmission,processing and service,this paper develops a framework for collecting and processing monitoring data.Based on the REST interface and Flume framework,original monitoring information was collected in real time,and then the buffer and persistent storage of the monitoring data flow were realized with Kafka platform.The monitoring data was first preprocessed and then monitoring indices were calculated on the Spark Streaming processing platform,followed by a series of processes including merging and compression of alarm events.A final warning was then produced for the operation and maintenance personnel.The experimental results show that the framework has the ability to collect and process massive monitoring data with high efficiency,fully meeting the needs of real-time monitoring of integrated meteorological operations in both timeliness and accuracy.

关 键 词：监视数据 接口 框架 流式计算平台 气象综合业务 

分 类 号：TP319[自动化与计算机技术—计算机软件与理论]