基于Spark的电力调度数据整合模型  

Power dispatch data integration model based on Spark

作　　者：曲朝阳[1] 陈贺新 胡可为 刘耀伟 独健鸿 QU Zhaoyang;CHEN Hexin;HU Kewei;LIU Yaowei;DU Jianhong(College of Information Engineering, Northeast Dianli University, Jilin, Jilin 132012, China;Control Center, State Grid Jilin Electric Power Co. , Ltd. , Changchun 130000, China;Jilin Power Supply Company, State Grid Jilin Electric Power Co. , Ltd. , Jilin, Jinlin 132000, China;Fengman Power Plant in Jilin City, Jilin, Jinlin 132113, China)

机构地区：[1]东北电力大学信息工程学院,吉林吉林132012 [2]国网吉林省电力有限公司调控中心,长春130000 [3]国网吉林省电力有限公司吉林供电公司,吉林吉林132000 [4]吉林市丰满发电厂,吉林吉林132113

出　　处：《计算机工程与应用》2017年第19期65-70,共6页Computer Engineering and Applications

基　　金：国家自然科学基金(No.51277023);吉林省科技发展计划项目(科技创新中心)(No.20160623004TC)

摘　　要：随着大数据理念在电力行业的应用,构建电力调度数据仓库是支撑电力调度中心统一数据平台的基础,针对电力调度中心的数据仓库将多源数据整合时面临的重复冗余和不一致问题,提出一种基于Spark的电力调度数据整合模型。设计并行化正向最大匹配去冗算法,对多个系统内冗余数据进行过滤操作;给出面向关联度的数据一致性处理方法,依据特征向量的夹角余弦值判断数据间的联系,进而对不一致数据修复。通过对某电力调度中心的数据进行整合实验,验证了该数据整合模型的可行性。With the concept of data applying in the electric power industry,the construction of electric power dispatchingdata warehouse is foundation to support the unified data platform in the power dispatching center,in view of the multisourcedata integration in the power dispatching center data warehouse there are redundant and inconsistent problems.This paper puts forward a kind of power dispatch data integration model based on Spark.First of all,the design of parallelforward maximum matching to remove redundancy algorithm,it filters the multiple system redundant data;then it givesdata consistency processing method of correlation degree-oriented.It judges link of data according to the angle value ofthe feature vector,and repairs the inconsistent data.Based on the data integration of electric power dispatching center,itverifies the practicability of the data integration model.

关 键 词：电力调度中心 关联度 特征向量 数据整合 Spark平台 

分 类 号：TP39[自动化与计算机技术—计算机应用技术]