数据同步技术在气象大数据云平台对外服务中的应用  

Application of Data Synchronization Technology in External Services of Meteorological Big Data Cloud Platform

作　　者：邱玲 宋智 吕爽 杨雪 QIU Ling;SONG Zhi;LYU Shuang;YANG Xue(Sichuan Meteorological Observation and Data Centre,Chengdu 610072,China;Heavy Rain and Drought-Flood Disasters in Plateau and Basin Key Laboratory of Sichuan Province,Chengdu 610072,China)

机构地区：[1]四川省气象探测数据中心,四川成都610072 [2]高原与盆地旱涝灾害四川省重点实验室,四川成都610072

出　　处：《计算机与现代化》2024年第7期76-81,共6页Computer and Modernization

基　　金：四川省科技计划基金资助项目(20227FS0544)。

摘　　要：目前,气象大数据云平台仅服务于气象内网用户,为进一步拓展平台的对外服务能力,满足气象行业用户对海量气象数据存储与共享应用需求,基于分布式存储和数据服务接口等方面关键技术,在单向网络安全隔离区(Demilitarized Zone,DMZ)设计一套安全、高效的气象大数据存储和服务体系。针对结构化气象数据,研究设计一种优化的基于数据库Binlog的数据同步方法;针对非结构化气象数据,研究设计一种优化的基于数据库Binlog+ETL机制相结合的数据同步方法,实现气象内网大数据云平台和DMZ区数据环境间的海量气象数据实时秒级同步。本文方法解决了气象数据对行业用户服务数据量少、共享服务能力差、安全性低等问题。应用实践表明,该数据同步技术延时在秒级以内且数据一致性良好,确保了气象大数据云平台对外服务的实时性和准确性。Currently,the meteorological big data cloud platform exclusively serves meteorological Intranet users.In order to fur⁃ther expand the platform’s capability for external services and meet the extensive demands of meteorological industry users for the storage and sharing of massive meteorological data applications,a secure and efficient meteorological big data storage and ser⁃vice system is designed based on key technologies such as distributed storage and data service interfaces.This system is imple⁃mented in the Unidirectional Network Security Isolation Zone(Demilitarized Zone,DMZ).For structured meteorological data,an optimized data synchronization method based on the Database Binlog in a distributed database is researched and designed.For unstructured meteorological data,an optimized data synchronization method combining the Database Binlog and ETL mecha⁃nisms is researched and designed to achieve real-time second-level unidirectional synchronization of massive meteorological data between the meteorological intranet big data cloud platform and the DMZ area.The method proposed in this paper addresses issues such as low data volume,poor sharing service capabilities,and low security for meteorological data in industry user ser⁃vices.Application practices demonstrate that the data synchronization technology achieves synchronization delays within seconds,maintains good data consistency,and ensures the real-time and accurate external service of the meteorological big data cloud platform.

关 键 词：数据同步 Binlog ETL 气象大数据 DMZ区 

分 类 号：TP311.13[自动化与计算机技术—计算机软件与理论]