Spark平台上利用网络加权Voronoi图的分散迭代社区聚类并行化研究  

DECENTRALIZED ITERATIVE COMMUNITY CLUSTERING PARALLELIZATION BASED ON NETWORK WEIGHTED VORONOI DIAGRAM ON SPARK PLATFORM

作　　者：颜烨[1] 张学文[2] 王立婧 Yan Ye;Zhang Xuewen;Wang Lijing(College of Electrical Information,City College of Science and Technology,Chongqing University,Chongqing 402167,China;Mechanical Engineering College,Beihua University,Jilin 132021,Jilin,China)

机构地区：[1]重庆大学城市科技学院电气信息学院,重庆402167 [2]北华大学机械工程学院,吉林吉林132021

出　　处：《计算机应用与软件》2021年第3期14-21,38,共9页Computer Applications and Software

基　　金：吉林省自然科学基金项目(20150101025JC);高档数控机床科学与基础制造装备科技重大专项(2015ZX040003002);重庆市本科高校大数据智能化类特色专业建设项目(渝教高发[2018]12号)。

摘　　要：针对当下数据大规模增长对计算能力需求的急剧增长,传统独立运行的机器在大规模网络社区中执行社区检测操作时无法提供所需的数据处理能力的问题,提出一种网络加权Voronoi图的并行分散迭代社区聚类法(NWVD-PDICCM)。利用基于网络加权Voronoi图的分散迭代社区聚类方法(NWVD-DICCM)提取大型网络的有效社区结构。结合并行聚类方法,将DICCM方法的操作从串行过程转换为并行计算。利用执行并行社区聚类时的图分区,通过最小化从属工作者之间的通信来加速该过程。仿真实验结果表明,NWVD-PDICCM可以与一系列计算机架构平台共同运行,并且实现基于Spark平台的并行操作,相比其他几种较新的方法,在大规模网络数据处理能力方面得到显著提升。In order to solve the problem that the demand for computing power is increasing rapidly due to the large-scale growth of data,and traditional independent machines can not provide the required data processing capability when performing community detection operations in large-scale network communities,a parallel decentralized iterative community clustering method based on network weighted Voronoi diagrams(NWVD-PDICCM)is proposed.The effective community structure of large-scale networks was extracted by using the decentralized iterative community clustering method based on weighted Voronoi graph(NWVD-DICCM).Combining with parallel clustering method,the operation of DICCM method was transformed from serial process to parallel computing.The process was accelerated by minimizing communication between subordinate workers using graph partitioning when performing parallel community clustering.The simulation results show that NWVD-PDICCM can run together with a series of computer architecture platforms and realize parallel operation based on Spark platform.Compared with other new methods,the proposed method has significantly improved the data processing capability of large-scale network.

关 键 词：大规模网络数据 网络加权Voronoi图 聚类社区 分散迭代 并行计算 Spark平台 

分 类 号：TP391[自动化与计算机技术—计算机应用技术]