基于随机森林的文本分类并行化  

Parallel Text Categorization of Random Forest

作　　者：彭徵 王灵矫[1] 郭华[1] PENG Zheng;WANG Ling-jiao;GUO Hua(The College of Information Engineering,Xiangtan University,Xiangtan,Hunan 411105,China)

机构地区：[1]湘潭大学信息工程学院,湖南湘潭411105

出　　处：《计算机科学》2018年第12期148-152,共5页Computer Science

摘　　要：文本分类是信息检索的核心技术。传统的文本分类系统由于单机的计算与存储能力有限,已经不适用于大数据时代。在Spark大数据平台上并行地运行算法对文本进行分类,以数据和任务的并行化来提高算法的效率具有现实性和紧迫性。文中提出了改进的不平衡数据随机森林算法,通过对训练样本的多数类进行欠取样且对少数类进行有放回取样从而形成新训练样本的方法来减少不平衡数据对随机森林的影响。实验结果表明,新算法在处理不平衡数据集上的少数类时提高了分类的正确率。Text categorization is one of the core technologies of information retrieval.Because of the limited computing performance and storage capacity in a computer,the traditional text categorization method can’t be suitable for big data era nowadays.It is realistic and urgent to execute algorithms for classifying the text in parallel to improve the efficiency of algorithm by the parallelization operation of data and tasks on the big data platform of Spark.This paper proposed an improved random fo-rest algorithm for the imbalanced data.It can reduce the impact of imbalanced data on random fo-rests by under-sampling the majority class samples and back-sampling the minority class samples to make up new trai-ning samples.The experimental results show that the new algorithm improves the categorization accuracy of the minority classes when handling imbalanced data sets.

关 键 词：文本分类 SPARK 随机森林 不平衡数据 并行化 

分 类 号：TP311[自动化与计算机技术—计算机软件与理论]