配电网监测大数据的Impala快速查询技术  

Fast query technology for monitoring big data of distribution network based on Impala

作　　者：屈志坚 陈鼎龙 巩奇 QU Zhi-jian;CHEN Ding-long;GONG Qi(School of Electrical and Automation Engineering,East China Jiaotong University,anchang 330013,China;Zhengzhou Rail Transit Co.Ltd.,Zhengzhou 450000,China)

机构地区：[1]华东交通大学电气与自动化工程学院,江西南昌330013 [2]郑州市轨道交通有限公司,河南郑州450000

出　　处：《电力科学与技术学报》2018年第2期148-156,共9页Journal of Electric Power Science And Technology

基　　金：国家自然科学基金(51267005;51567008);江西省自然科学基金(20161BAB206156);江西省杰出青年人才计划项目(20162BCB23045)

摘　　要：针对目前配电网监测大数据SQL交互查询速度慢的问题,对配电网监测数据类型进行归类整理,利用Impala分布式处理工具重点研究一种监测大数据的MPP快速查询技术。通过协调节点将查询计划解析为执行计划树,将计划树的片段分配至多个从节点并行执行,各从节点将中间结果按执行计划树流式传递回协调节点,再通过多机集群的全内存并行执行加速查询。选用四机监控系统集群为例进行加载测试和查询性能测试,结果表明:相较关系数据库,MPP大数据快速查询技术大幅提高了数据加载速度。对北京某动车段配电监测的千万级数据记录,关系数据库和Hive数据仓库至少都需94s以上,而MPP快速查询仅需约320ms,查询性能提升近3个数量级,大幅提高了监测大数据的查询处理速度。Regarding to the low speed of SQL interactive query for monitoring big data in distribution network,the type of monitoring data is classified and an MPP fast query technology is proposed by using impala which is a distributed processing tool.Firstly,the query plan is parsed into an execution plan tree by the coordinate node.Next,the fragments of plan tree are delivered to multiple slave nodes and executed in parallel.Then,the results are steaming transferred from slave nodes to coordinate nodes according to the plan tree.Finally,the query speed is accelerated by performing the full memory of a machine cluster parallely.Besides,a monitoring system composed of four computers is simulated to verify this technology.It is shown that the loading speed of MPP big data querying technology is much faster than the relational database.For ten millions monitoring data records of a Beijing Motor Car Depot,MPP query technology only needs about 320 ms,while relational database and hive data warehouse need at least 94 s.The query processing speed of monitoring big data is well improved.

关 键 词：配电网大数据 分布式存储 IMPALA MPP 快速查询 

分 类 号：TM73[电气工程—电力系统及自动化] TP274[自动化与计算机技术—检测技术与自动化装置]