基于Hive的高可用双引擎数据仓库  

High Availability Dual Engine Data Warehouse Based on Hive

作　　者：李翀[1] 张彤彤 杜伟静 刘学敏[1] LI Chong;ZHANG Tong-Tong;DU Wei-Jing;LIU Xue-Min(Computer Network Information Center,Chinese Academy of Sciences,Beijing 100190,China;University of Chinese Academy of Sciences,Beijing 100190,China)

机构地区：[1]中国科学院计算机网络信息中心,北京100190 [2]中国科学院大学,北京100190

出　　处：《计算机系统应用》2019年第9期65-71,共7页Computer Systems & Applications

基　　金：中国科学院"十三五"信息化专项(XXH13504-03)~~

摘　　要：打破信息孤岛,整合异构数据,汇聚共享交换,深度分析挖掘,提供行业领域辅助决策和态势分析具有深远的理论和应用价值.本文以中国科学院教育科研态势感知服务的实际需求为牵引,设计并实现了一套基于Hive的Hadoop/Spark双计算引擎大数据仓库,支持多种方式OLAP分析,进行了可用性、负载均衡、资源管理的优化设计,为后续进行全院数据汇聚挖掘、知识图谱构建、学科态势分析提供了平台支撑.实验表明,系统灵活高效,高可用可扩展,资源调度科学,负载均衡效果明显.Breaking isolated information island,integrating heterogeneous data,gathering and sharing exchanges,conducting in-depth analysis and mining,and providing industry-side decision-making and situation analysis have farreaching theoretical and applied value.Based on the actual demand of the situational awareness service of the Chinese Academy of Sciences,this study designs and implements a Hive-based Hadoop/Spark dual computing engine big data warehouse supporting OLAP analysis in multiple ways,and carries out an optimization design of usability,load balancing,and resource management,which provides platform support for the subsequent data aggregation and mining,knowledge map construction and discipline situation analysis.Experimental results show that the system is flexible,efficient,available,and scalable,the resource scheduling is scientific,and the load balancing effect is obvious.

关 键 词：数据仓库 Hive 高可用 OLAP HADOOP 

分 类 号：TP3[自动化与计算机技术—计算机科学与技术]