数据同步技术在气象大数据云平台中的应用  

Application of Data Synchronization Technology in CMADaaS

作　　者：冯勇 吕冠儒 李微 安浩然 Feng Yong;Lv Guanru;Li Wei;An Haoran(Key Laboratory for Meteorological Disaster Prevention and Mitigation of Shandong,Ji'nan 250031,China;Shandong Meteorological Information Center,Ji'nan 250031,China;Hunan Meteorological Information Center,Changsha 410118,China;Jinan High-Tech Industrial Development Zone Bureau of Meteorology,Ji'nan 250031,China)

机构地区：[1]山东省气象局气象防灾减灾重点实验室,山东济南250031 [2]山东省气象信息中心,山东济南250031 [3]湖南省气象信息中心,湖南长沙410118 [4]济南高新技术产业开发区气象局,山东济南250031

出　　处：《科学技术创新》2022年第21期96-99,共4页Scientific and Technological Innovation

基　　金：基于“天擎”的气象数据可视化服务关键技术研究(山东省气象局重点课题2021sdqxz06);基于“天擎”的机器学习平台关键技术研究与开发-周雪松(山东省气象局重点课题2021sdqxz02);团队资助:山东省精准预报技术创新团队、山东台风与海洋气象创新团队共同资助。

摘　　要：为满足山东省气象预报、公共服务等气象业务对海量气象数据访问需求和毫秒级响应的时效需求,基于数据库变更日志Binlog和分布式消息服务Kafka技术,在省级气象大数据云平台中设计了一套气象数据同步框架,并通过存储管理系统(SOD)进行策略配置,解决了原有CIMISS数据环境在规模和性能上的服务瓶颈。实践证明:基于Binlog+Kafka的气象数据同步效率可达110000条/s,耗时延迟毫秒级,有效支撑了全省气象业务系统对海量气象数据的时效需求。To meet the demand for massive meteorological data access and millisecond response timeliness for meteorological services such as meteorological forecasting and public services in Shandong Province,a meteorological data synchronization framework is designed in the provincial meteorological big data cloud platform based on Binlog,a database change log,and Kafka,a distributed messaging service technology,and policy configuration through storage management system(SOD),which solves the original CIMISS data environment in the scale and performance of the service bottleneck.It is proved that the efficiency of weather data synchronization based on Binlog+Kafka can reach 110,000 items/s with milliseconds delay,which effectively supports the province's meteorological business system's demand for massive weather data timeliness.

关 键 词：Binlog Kafka 气象大数据云平台 数据同步 存储管理 

分 类 号：TP311[自动化与计算机技术—计算机软件与理论]