Spark框架下分布式K-means算法优化方法  

Optimization method of distributed K-means algorithm based on Spark

作　　者：王法玉[1] 刘志强 WANG Fa-yu;LIU Zhi-qiang(Key Laboratory of Tianjin City of Intelligent Computing and Software New Technology,Tianjin University of Technology, Tianjin 300384, China)

机构地区：[1]天津理工大学智能计算及软件新技术天津市重点实验室

出　　处：《计算机工程与设计》2019年第6期1595-1600,共6页Computer Engineering and Design

基　　金：国家自然科学基金项目(61571328);天津市自然科学基金项目(15JCTPJC60100);天津市普通高等学校本科教学质量与教学改革研究计划基金项目(171006008F)

摘　　要：针对传统K-means算法在处理海量数据时存在计算效率低和时间复杂度高的缺点,提出一种基于Spark计算框架的改进K-means算法。利用网格单元保存数据点的空间位置信息,通过与聚类中心的空间位置关系减少冗余计算,为提高算法处理海量数据的能力,采用Spark框架对算法进行并行化实现。在集群环境下进行测试,基于Spark框架的改进后算法能有效降低计算的时间复杂度,算法具有良好扩展性,计算效率有显著提高。To solve the disadvantages of low computational efficiency and high time complexity of traditional K-means algorithm when dealing with massive data, an improved K-means algorithm was proposed based on Spark computing framework. The grid cell was used to save spatial position information of the data points, and redundant calculations were reduced through the spatial relationship with clustering centers. To improve the algorithm ’s ability to process massive data, the Spark framework was used to parallelize the algorithm. By testing in a cluster environment, the improved algorithm based on the Spark framework can effectively reduce the computational time complexity, the algorithm shows good scalability, and the computational efficiency is significantly improved.

关 键 词：K-MEANS算法 Spark计算框架 分布式 网格 空间位置 

分 类 号：TP301.6[自动化与计算机技术—计算机系统结构]