基于Kudu的大数据平台实时业务处理能力提升方案  

Scheme of enhancing real-time business processing capabilities based on Kudu for the big data platform

作　　者：顾飞杨 孔莹 GU Feiyang;KONG Ying(Shanghai Branch of China Telecom Group Co.,Ltd.,Shanghai 200042,China)

机构地区：[1]中国电信股份有限公司上海分公司

出　　处：《电信科学》2019年第10期151-156,共6页Telecommunications Science

摘　　要：针对目前Hadoop大数据平台实时业务处理能力较差的难点,研究了国际最先进的Kudu列存储作为HDFS块存储的有效补充的理论,阐述了利用Kudu和Spark提供的主键索引和内存加速,有效解决大数据平台无法支持实时入库、增量更新和SQL关联查询等业务痛点的技术实现方法。实验效果证明了方法对提升大数据平台实时业务处理能力的作用。Aiming at the difficulty of real-time business processing capability of Hadoop big data platform,the theory of the most advanced Kudu column storage was studied as an effective complement of HDFS block storage,and the use of primary key index and memory acceleration provided by Kudu and Spark was illustrated to effectively solve the big data platform cannot support the technical implementation methods of business pain points such as real-time warehousing,incremental update and SQL-related query.The experimental results prove the effect of the method on improving the real-time business processing capability of the big data platform.

关 键 词：Kudu 大数据 列存储 主键索引 内存加速 实时入库 增量更新 SQL关联查询 

分 类 号：TP311.13[自动化与计算机技术—计算机软件与理论]