大数据下基于MapReduce的Dirichlet朴素贝叶斯文本分类算法  

MapReduce-based Dirichlet Naive Bayes Text Classification Algorithm Under Big Data

作　　者：胡晓东[1] 高嘉伟[2] Hu Xiaodong;Gao Jiawei(Department of Electronic and Information Engineering,Shanxi Institute of Economic Management,Taiyuan 030024,China;School of Computer and InformationTechnology,Shanxi University,Taiyuan 030006,China)

机构地区：[1]山西经济管理干部学院电子信息工程系,太原030024 [2]山西大学计算机与信息技术学院,太原030006

出　　处：《科技通报》2017年第9期124-129,共6页Bulletin of Science and Technology

基　　金：国家自然科学基金(No.61202018)

摘　　要：针对当前大数据环境下朴素贝叶斯文本分类算法在处理文本分类时存在的数据稀疏、分类不准及效率低的问题,本文提出了一种基于MapReduce的Dirichlet朴素贝叶斯文本分类算法。算法首先根据体征词语义因素以及类内分布情况对权重进行加权调整,以此对的计算公式进行修正;引入了统计语言建模技术中的Dirichlet数据平滑方法来降低数据稀疏对分类性能的影响,并在Hadoop云计算平台采用MapReduce编程模型实现本文算法的并行化。通过测试实验对比分析可知,本文算法显著提高了传统朴素贝叶斯文本分类算法的准确率、召回率,并具有优良的可扩展性和大数据处理能力。According to the data environment of naive Bayes text classification algorithm in textclassification of the existence of data sparse and inaccurate classification and low efficiency problem,this paper proposes a Dirichlet naive Bayes text classification algorithm based on MapReduce.Firstly,according to the words and signs within the meaning of the factors and the distribution of the weightclasses is adjusted to be corrected on the TF-IDF;which introduces statistical language modelingtechniques Dirichlet data smoothing methods to reduce the impact on the classification performance ofthe sparse data,in the Hadoop cloud computing platform,the MapReduce programming model is used torealize the algorithm of this paper.Through the experiment analysis shows that the algorithm significantlyimproves the traditional naive Bayes text classification algorithm accuracy rate and recall rate,and hasexcellent expansibility and data processing ability.

关 键 词：朴素贝叶斯 文本分类 TF-IDF修正 Dirichlet数据平滑 MapReduce并行化 

分 类 号：TP311[自动化与计算机技术—计算机软件与理论]