基于Spark的极限学习机算法并行化研究  

Study of ELM Algorithm Parallelization Based on Spark

作　　者：刘鹏[1,2] 王学奎 黄宜华[4] 孟磊 丁恩杰[1,2,3] LIU Peng;WANG XUe-kui;HUANG Yi-hua;MENG Lei;DING En-jie(Internet of Things Perception Mine Research Centre,China University of Mining and Technology,Xuzhou221008,China;National and Local Joint Engineering Laboratory of Internet Application Technology on Mine,Xuzhou221008,China;Schoo1 of Information and Control Engineering,China University of Mining and Technology,Xuzhou221116,China;PASA Big-data Laboratory.Department of Computer Science,Nanjing University,Nanjing210023,China)

机构地区：[1]中国矿业大学物联网(感知矿山)研究中心,徐州221008 [2]矿山互联网应用技术国家地方联合工程实验室,徐州221008 [3]中国矿业大学信息与控制工程学院,徐州221116 [4]南京大学计算机系PASA大数据实验室,南京210023

出　　处：《计算机科学》2017年第12期33-37,共5页Computer Science

基　　金：国家重点研发计划:矿山安全生产物联网关键技术与装备研发(2017YFC0804400;2017YFC0804401);国家自然科学基金项目(61471361;41302203)资助

摘　　要：极限学习机算法虽然训练速度较快,但包含了大量矩阵运算,因此其在面对大数据量时,处理效率依然缓慢。在充分研究Spark分布式数据集并行计算机制的基础上,设计了核心环节矩阵乘法的并行计算方案,并对基于Spark的极限学习机并行化算法进行了设计与实现。为方便性能比较,同时实现了基于Hadoop MapReduce的极限学习机并行化算法。实验结果表明,基于Spark的极限学习机并行化算法相比于Hadoop MapReduce版本的运行时间明显缩短,而且若处理数据量越大,Spark在效率方面的优势就越明显。Extreme learning mechine(ELM)has high training speed,but with lots of matrix operations,it rernams poor efficiency while applied to massive amount of data.After thorough research on parallel computation of Spark resilient distributed dataset(RDD),we proposed and implemented a parallelized algorithm of ELM based on Spark.And for convenienceof performance comparison,Hadoop-MapReduce-based version was also implemented.Experimental results show that the training efficiency of the Spark-based ELM parallelization algorithm is significantly improved than the Hadoop-MapReduce-based version.If the amount of data processed is greater,the advantage of Spark in efficiency is more obvious.

关 键 词：限学习机 并行化 SPARK RDD Hadoop MAPREDUCE 

分 类 号：TP18[自动化与计算机技术—控制理论与控制工程]