基于Hadoop的海量气象水文数据并发处理模型  

Concurrent processing model of massive meteorological hydrological data based on Hadoop

作　　者：李辉 王建文 叶明雯 LI Hui;WANG Jianwen;YE Mingwen(Troops 61741 PLA,Beijing 100094,China;National Defence University,Beijing 100091,China;Troops 31010 PLA,Beijing 100081,China)

机构地区：[1]解放军61741部队,北京100094 [2]国防大学,北京100091 [3]解放军31010部队,北京100081

出　　处：《计算机应用》2018年第A02期187-191,205,共6页journal of Computer Applications

摘　　要：针对关系型数据库以及单节点处理难以满足海量气象水文数据存储与处理的问题,提出了一种基于Hadoop的海量气象水文数据并发处理模型(CPHDH)。该模型结合云计算技术,利用计算机集群以及HDFS文件系统实现海量气象水文数据的分布式存储;同时,基于MapReduce编程框架完成海量气象水文数据的高效并行处理。仿真实验中,CPHDH能够有效利用Hadoop平台的分布式存储和并行处理框架实现海量降雨要素信息的高效存储和处理,相比单机运行基于遗传算法的气象观测数据区间值属性约简算法(MOIvGA)进行降水影响因子属性约简,CPHDH中的属性子集对无雨、小雨和中雨预测效率分别提高了10. 74%、6. 19%、4. 42%。To solve the problem that relational database and single node processing is difficult to store and process mass meteorological hydrological data,a Concurrent Processing model of massive meteorological Hydrological Data Based on Hadoop(CPHDH)was proposed.By using cloud computing technology and HDFS file system,mass meteorological hydrological data could be stored distributedly;by using MapReduce framework,massive meteorological hydrological data was processedefficiently.In simulation,CPHDH effectively used Hadoop platform s distributed storage and processing framework to achieve efficient storage of massive rainfall information and attribute reduction processing;compared with the single-machine MOIvGA(attribute reduction algorithm for Meteorological Observation data Interval value based on Genetic Algorithm)for precipitation influence factor attribute reduction,the prediction efficiency attribute subset in CPHDH increases by 10.74%,6.19%and 4.42%respectively in the prediction of no rain,light rain and moderate rain.

关 键 词：海量气象水文数据 HADOOP 分布式文件存储系统 MAPREDUCE 并发处理 

分 类 号：TP311[自动化与计算机技术—计算机软件与理论]