基于数据感知的气象算法调度框架设计与应用  

Design and Application of Meteorological Algorithm Scheduling Framework Based on Data Perception Technology

作　　者：霍庆 何文春[1] 高峰[1] 陈士旺 徐拥军[1] 王琦[1] Huo Qing;He Wenchun;Gao Feng;Chen Shiwang;Xu Yongjun;Wang Qi(National Meteorological Information Center,Beijing 100081)

机构地区：[1]国家气象信息中心,北京100081

出　　处：《应用气象学报》2024年第4期502-512,共11页Journal of Applied Meteorological Science

基　　金：中国气象局重点创新团队(CMA2024ZD05);中国国家铁路集团有限公司科研计划课题(N2022T011);国家气象信息中心创新团队(NMIC-2024-ZD11)。

摘　　要：气象产品加工任务往往由不同算法组合而成,其状态对数据源具有高依赖性。高效感知数据状态,并及时合理启动调度,对提升气象产品加工的时效性至关重要。设计并建设了基于数据感知技术的气象算法调度框架,对比测试结果分析表明:在实时分析数据源到报状态的基础上,数据感知调度实现数据到报即算法启动运行,任务调度延迟较定时调度显著降低,由3784 ms降至11 ms。数据感知调度作为气象大数据云平台加工流水线的核心能力,已在国省部署并实时运行。目前在国家级业务中支撑19种核心业务算法的高效调度,日调度次数约为6.67×10^(5),平均调度延迟为31 ms;在省级业务中支撑14种算法的高效调度,日调度次数约为8×10^(4),平均调度延迟为156 ms。此外,数据感知调度实现了气象业务上下游算法的无缝串联,可为解决气象业务间的脱节问题、提升气象业务间的协同性提供解决方案。The generation efficiency of data products depends on both the computational efficiency and the startup efficiency of algorithms.In meteorological operations,algorithms are typically data-driven,meaning they are initiated immediately upon data arrival to accelerate the generation time of data products.Therefore,data-driven meteorological services urgently require an efficient task scheduling framework to achieve the goal of starting and running algorithms as soon as data arrive,and to improve the generation efficiency of meteorological data products.CMA Big Data and Cloud Platform,referred to as Tianqing and led by National Meteorological Information Center began nationwide business operations in December 2021.The data processing line(DPL),as the core function of Tianqing,enables the unified management and centralized scheduling of meteorological algorithms.DPL has established various task scheduling capabilities,including timer-triggered scheduling,sequential scheduling,data-arrival scheduling,and manual scheduling.Among these methods,data-arrival scheduling based on data reporting status realizes the algorithm to start immediately after data reporting,greatly improving the startup time of meteorological algorithms and the generation time of meteorological data products.Core functions of data-arrival scheduling include data state awareness components,task scheduling execution components,task scheduling post-processing components,and configuration management.Among these components,the data state perception component realizes real-time analysis of the reporting status of various meteorological data in the sky engine,and sends scheduling messages to the task scheduling execution component when the data reporting rate meets scheduling requirements.The task scheduling execution component combines the necessary resource information for the algorithm and computing nodes optimization,generate task scheduling instructions,and implement algorithm startup execution.The postprocessing phase of task scheduling includes gathe

关 键 词：数据感知技术 任务调度 气象算法 气象大数据云平台 加工流水线 

分 类 号：P409[天文地球—大气科学及气象学] TP393.09[自动化与计算机技术—计算机应用技术]