Hadoop在气象数据密集型处理领域中的应用  

Application of Hadoop in Data-Intensive Processing of Meteorological Data

作　　者：肖卫青[1] 杨润芝[1] 胡开喜[1] 林润生[1] 刘立明[1] 谷军霞[1] 

机构地区：[1]国家气象信息中心,北京100081

出　　处：《气象科技》2015年第5期823-828,共6页Meteorological Science and Technology

摘　　要：气象资料的统计分析计算属于数据密集型计算,目前的处理方式多为单机处理,对大量数据的处理比较慢,难以应对日益增长的数据,对气象资料的研究形成一定的制约。针对数据密集型气象数据的处理,尝试应用Hadoop的MapReduce思想提高计算效率;对Hadoop在处理大量小文件组成的气象数据时的低效率,提出对原始文件进行预处理,将多个小文件整合成能直接用于计算的大文件。试验证明,该方法解决了Hadoop处理大量小文件时的低效率问题,通过与Oracle入库检索的比较,应用Hadoop处理数据密集型气象资料具有实际意义。The statistical analysis of meteorological data processing is data- a single machine. The speed is too slow when the data set is large intensive and always conducts on which restrains researches of meteorological data. Hadoop and MapReduce are used to speed up the data-intensive processing of meteorological data. In allusion to the low efficiency of processing enormous and small files by using Hadoop, a preprocess is conducted to integrate the enormous and small files to a large one. The experiment proved that this method can solve the low efficiency problem when using Hadoop to process enormous and small files. Comparing with Oracle, it is more useful to use Hadoop to process data- intensive Meteorological computing.

关 键 词：HADOOP HDFS MAPREDUCE 气象数据 数据密集型计算 

分 类 号：P409[天文地球—大气科学及气象学]