基于HDFS和IMPALA的碰撞比对分析  

Analysis of Collision Alignment Based on HDFS and IMPALA

作　　者：王艳[1] 潘晨光[1] 

机构地区：[1]公安部第一研究所,北京100048

出　　处：《电视技术》2015年第14期94-98,共5页Video Engineering

基　　金："十二五"国家科技支撑计划项目(2011BAK05B06;2011BAK05B05)

摘　　要：处理公共安全大数据需实现查询、全文检索、即时数据推送和内存计算等实时业务分析,而基于Hadoop生态系统建立的Hive不支持海量数据的实时查询,无法实现交互式查询和报表查询快速返回数据结果。针对Hive实时查询的不足,首次将Impala架构应用于公共安全数据处理上,结合Impala实时查询引擎的原理,分析了Map Reduce上Hive的实时查询缺陷,比较了Impala和Hive的实时性能,改进了Impala上运行CURE碰撞比对算法,实现了异地数据碰撞比对的分析与展现。本系统的方案在实战应用中取得了较好的效果,可为公安安全领域构建大规模数据处理与分析系统提供技术参考。Real-time service analysis must realize the function of query, full text retrieval, real-time data push and memory computing to process big data of public safety. But Hive based on Hadoop don＇t support query real-time big data. The new method of Impala query engine to process the police data is proposed to solve real-time query of Hive. The defects of Hive real-time query based on MapReduce are analyzed. The principle of Impala query engine is presented which is compared to the performance of Impala. The system is designed by improving the CURE algorithm on Impala to show the visualization of collision on long-distance data. Finally, the effectiveness of the system is validated by the case of query.

关 键 词：公共安全 数据处理 HDFS MAPREDUCE Hive 

分 类 号：TP311[自动化与计算机技术—计算机软件与理论]