基于Hadoop的海洋数据存储平台设计与实现  

Design and implementation on storage platform of marine data based on Hadoop

作　　者：宋晓[1] 韩璐遥 韦广昊[1] 杨锦坤[1] 郑兵 SONG Xiao;HAN Luyao;WEI Guanghao;YANG Jinkun;ZHENG Bing(National Marine Data and Information Service,Tianjin 300171,China)

机构地区：[1]国家海洋信息中心,天津300171

出　　处：《海洋信息技术与应用》2023年第2期101-107,共7页JOURNAL OF MARINE INFORMATION TECHNOLOGY AND APPLICATION

基　　金：国家重点研发计划重点专项(2022YFC3105100,2021YFC3101600)。

摘　　要：海洋数据具有多来源、多尺度、多频度等复杂特性,传统事务型数据库无法满足海量非结构化数据在存储管理、关联查询、复杂分析等方面的需求。针对这一问题,本文采用HDFS+MapReduce+HBase的分布式数据存储架构,利用数据编码压缩存储技术,实现了数据分布式存储管理和处理,并提出了一种多级索引存储管理技术以实现数据快速关联查询;开发了基于WebService的数据查询接口、海洋环境数据信息共享接口、决策支持数据服务接口、智能分析数据服务接口等应用接口。实验表明,对比传统技术架构,该平台在非结构化海洋数据的数据存储管理、数据处理和业务应用等方面具有一定优势。Marine data have features of multi-source,multi-scale and multi-frequency,the transactional database can't meet the requirements of unstructured data management,correlation query and complex analysis.To solve this problem,using distributed storage architecture of HDFS+MapReduce+HBase and Data coding technology to realize data distributed storage management and processing;Proposed a multi-level index technology to achieve rapid data query;Implementation of data query based on WebService interfaces,Marine data information sharing interface,decision support data service interface,intelligent analysis data service interface application interface,etc.Experiments show that,compared to traditional technology architecture,the platform has certain advantages in data management,data processing and business application of unstructured marine data.

关 键 词：Hadoop体系架构 多级索引 数据编码 海洋数据 

分 类 号：P717[天文地球—海洋科学] TP392[自动化与计算机技术—计算机应用技术]