基于MapReduce的特征选择并行化研究  

MapReduce Based Feature Selection Parallelization

作　　者：陆江[1] 李云[1] 

机构地区：[1]南京邮电大学计算机学院,南京210023

出　　处：《计算机科学》2015年第8期44-47,81,共5页Computer Science

基　　金：江苏省自然科学基金(BK20131378);国家自然科学基金(61105082)资助

摘　　要：特征选择已经成为一种对高维数据进行预处理的必不可少的手段。随着数据规模的爆炸性增长,传统的特征选择算法已经不能满足当前高维大规模数据的处理要求。采用Google的MapReduce编程模型,设计了一种分布式的基于局部学习的特征选择算法D-logsf。在多个现实和合成数据集上的实验表明,分布式特征选择算法D-logsf具有较好的可靠性,且与传统特征选择算法Logsf相比可以获得接近线性的加速比,同时可以有效处理大规模数据集。Feature selection has become a necessary preprocessing procedure for high-dimensional data With the explosive growth of data size, the traditional feature selection algorithm can not meet the current requirements of processing large-scale and high-dimensional data. Resorting to Google＇ s MapReduce programming model, we designed a distributed local learning-based feature selection algorithm D-logsf. Experiments were conducted on several real and synthesis data sets. The results show that the D-logsf algorithm is correct and has good reliability. Compared with traditional feature selection algorithm Logsf, D-logsf can obtain approximate linear speedup. Moreover, D-logsf can effectively handle large- scale data set.

关 键 词：特征选择 局部学习 分布式 MAPREDUCE 

分 类 号：TP391[自动化与计算机技术—计算机应用技术]