Hadoop在气象数据存储中的应用  

The application of Hadoop in the storage of meteorological data

作　　者：任晓炜[1] 黄志[1] 詹利群[1] Ren Xiaowei;Huang Zhi;Zhan Liqun(Guangxi Regional Meteorological Information Center,Nanning Guangxi 530022)

机构地区：[1]广西区气象信息中心,广西南宁530022

出　　处：《气象研究与应用》2019年第1期65-68,共4页Journal of Meteorological Research and Application

基　　金：国家档案局科技项目(编号:2016-X-06)资助

摘　　要：利用虚拟资源池搭建基于Hadoop的大数据存储架构,将海量自动站文本数据、数字化历史图片以及二进制雷达基数据按照自定义ETL存储规则进行数据清洗之后存入大数据框架,在并发读取效率测试中取得了良好的效果,为应对海量气象资料增长在扩展性和系统性能方面提出的挑战提供解决思路和基本模型。A large data storage architecture based on Hadoop is constructed by using virtual resource pool. The massive text data,digitized historical pictures and binary radar-based data are cleaned according to the customized ETL storage rules and stored in the large data frame. Good results have been achieved in the concurrent reading efficiency test,which provides solutions and basic models to meet the challenges posed by the growth of mass meteorological data in terms of scalability and system performance.

关 键 词：大数据 HADOOP ETL 存储 

分 类 号：P49[天文地球—大气科学及气象学]