基于MapReduce模型的并行粒子群分簇算法研究  

Research on parallel particle swarm optimization clustering algorithm based on MapReduce model

作　　者：赵彦 孙俊[1] ZHAO Yan;SUN Jun(International Joint Laboratory of Pattern Recognition and Artificial Intelligence,Jiangnan University,Wuxi 214122,China;School of Internet of Things Engineering,Jiangsu Vocational College of Information Technology,Wuxi 214153,China)

机构地区：[1]江南大学人工智能与模式识别国际联合实验室,江苏无锡214122 [2]江苏信息职业技术学院物联网工程学院,江苏无锡214153

出　　处：《现代电子技术》2021年第7期137-141,共5页Modern Electronics Technique

基　　金：国家自然科学基金(61672263);江苏省自然科学基金(BK20131097);江苏省高职院校教师专业带头人高端研修(个人访学研修)基金项目(2019GRGDYX015);2017年江苏高校“青蓝工程”基金资助项目(2017JSJW007);江苏省第五期“333工程”第三层次培养对象基金资助项目(苏人才办〔2018〕6号);学院科研课题(JSITKY201804)。

摘　　要：在超大规模数据集的分簇管理上,存在大数据获取、存储、检索、分析和可视化等困难。面对爆炸式增长的数据,利用分布式、并行计算原理,在MapReduce模型的基础上构建并行粒子群优化算法(PSOC-MR),实现对超大规模数据的有效分簇处理。实验结果表明,PSOC-MR算法在集群节点数量与数据集大小等比例增加的情况下呈现良好的可扩展性,能在保持分簇质量的同时呈现线性加速,该算法可有效解决超大规模数据集的分簇问题,实现低成本、高性能的商用大数据分析。There are some difficulties in the clustering management of the data sets with super large scale,such as big data obtaining,storing,retrieving,analyzing and visualizing.Under the background of the explosive growth of data,a paralleled particle swarm optimization clustering algorithm based on MapReduce(PSOC-MR)is constructed according to the distributed principle and paralleled computing principle,so as to achieve effective clustering processing of the data with super large scale.The experimental results indicate that the PSOC-MR shows good scalability when the number of cluster nodes and the size of the data set increase with equal proportion,and can show linear acceleration while keeping due clustering quality.Therefore,the proposed algorithm can effectively cope with the difficulties in the clustering of the super large scale data sets,and achieve low-cost and high-performance analysis for commercial big data.

关 键 词：粒子群算法 分簇算法 并行计算 MAPREDUCE模型 分簇处理 大数据分析 

分 类 号：TN911.1-34[电子电信—通信与信息系统] TP391[电子电信—信息与通信工程]