地面气象资料统计处理系统设计与实现  

Design and Implementation of Surface Meteorological Data Statistical Processing System

作　　者：孙超[1] 霍庆 任芝花[1] 刘振[1] 肖卫青[1] 徐拥军[1] Sun Chao;Huo Qing;Ren Zhihua;Liu Zhen;Xiao Weiqing;Xu Yongjun(National Meteorological Information Center,Beijing 100081)

机构地区：[1]国家气象信息中心,北京100081

出　　处：《应用气象学报》2018年第5期630-640,共11页Journal of Applied Meteorological Science

基　　金：公益性行业(气象)科研专项(GYIIY201406016);国家气象信息中心"山洪地质灾害防治气象保障工程2016年建设"项目

摘　　要：为实现6万余个地面气象站资料实时统计处理,设计并实现了地面气象资料统计处理系统。系统采用灵活易扩展的技术框架,应用大数据分布式流处理技术实现高效的数据处理,较传统统计处理框架统计时效提升10掊以上,主要功能包括基于任务调度的定时统计、针对迟到数据和数据更正信息的统计结果滚动更新、自定义统计、统计处理通用算法服务等,采用国家级一级部署、各省同步应用的业务布局保证数据的一致性。2017年1月投入业务运行后,实时计算800余个多种尺度的统计项并通过全国综合气象信息共享平台(CIMISS)数据统一服务接口提供统计产品服务。分析表明:地面气象资料统计产品2017年月平均下载次数达到1951.4万次,在CIMISS所有400余种观测资料或产品中排名第三,为天气监测预报预警、气象决策服务、气候监测业务、公共气象服务等提供重要的基础数据支撑。Statistical products of surface meteorological data（SMD） are among the most-frequently-used data in meteorological research and operations. As the improvement of surface meteorological observation system over China, statistics of SMD have encountered problems such as large number of sites, wide variety of elements,and complexity of statistical strategy. With typical features of big data, it＇s possible for SMD to serve more precise and efficient operations nowadays, which is obviously beyond the capability of traditional serial processing framework.Aiming at precise and efficient statistic processing of data from more than 60000 surface weather stations, a statistical processing system for SMD is built based on big data technology. Compared to traditional serial processing framework, efficiency of the system has increased by more than 10 times and more statistics and function are provided, such as fast calculation, rolling update of statistical values according to late-arriving data and corrected information, and arbitrary time scale statistics. Storm distributed flow processing technology is applied in the system to realize efficient statistical calculations. Big data message transmission and cache technology are also applied to ensure the system＇s high efficiency and stability.Modular design framework ensures strong extensibility of the system, based on which statistics, quality control and evaluation algorithms are extended to varieties of data, e. g., upper-air, radiation, oceanic and aircraft measurements. The system is deployed at national meteorology department and its products are synchronously applied at the provincial level, for this layout ensures data consistency.The system is incorporated into China Integrated Meteorological Information Sharing System（CIMISS） and become its core data processing framework. The system provides more than 800 real-time multi-scale SMD statistical values to serve meteorological users and the public through CIMISS data unified service interface since January 201

关 键 词：地面气象资料 统计处理 分布式计算 模块化 滚动更新 

分 类 号：P413[天文地球—大气科学及气象学]