基于HBase+ ElasticSearch的海量交通数据实时存取方案设计  

Design scheme of massive traffic data real-time access based on HBase and Elastic Search

作　　者：董长青 任女尔 张庆余 田玉靖 

机构地区：[1]北京卡达克数据技术中心软件业务本部,天津300300

出　　处：《大数据》2017年第1期80-89,共10页Big Data Research

摘　　要：交通流数据具有数据海量、存储和交互速率快等特征,因此其数据的采集、存储及检索成为了车辆远程监控平台中的关键问题。采用LVS集群技术进行数据采集负载均衡,队列缓存处理I/O时延,HBase进行分布式数据存储;针对Hadoop实时在线数据处理不足的问题,整合Elastic Search并构建了分层索引。通过关键技术的设计和实现,车辆监控由400辆扩展到上万辆,PB级数据在线查询速度提升了10~20倍,验证了方案的高效性。Traffic data has the characteristics of massive and real-time, and its massive data acquisition, storage and retrieval has become a key issue in the vehicle remote monitoring platform. According to the study of these problems, the cluster technology of LVS was used to solve the data acquisition load balance, the queue cache model was used to solve I/O delay, and HBase distributed data storage scheme was used to solve the massive data storage. HBase integration ElasticS earch, which was aimed to solve the real-time online data processing problems of Hadoop, was designed to build a hierarchical index. Through the design and implementation of the key technologies, the number of vehicle monitoring had been promoted from 400 to 1 million, online query speed increased about 10 to 20 times based on PB level data. The results verified the efficiency of the scheme.

关 键 词：Hadoop/HBase ELASTIC Search LINUX虚拟服务器 海量数据 实时 

分 类 号：TP311[自动化与计算机技术—计算机软件与理论]