基于MapReduce的并行LAD模型评论主题提取算法研究  

Research on topic extraction algorithm based on MapReduce parallel LAD model

作　　者：薛行贵 高见文 张伯虎[1] 黄立勤[2] XUE Xinggui GAO Jianwen ZHANG Bohu HUANG Liqin(Graduate Management Unit, Engineering University of CAPF, Xi＇ an, Shanxi 710086, China College of Physics and Information, Fuzhou University, Fuzhou, Fujian 350116, China)

机构地区：[1]武警工程大学研究生管理大队,陕西西安710086 [2]福州大学物理与信息工程学院,福建福州350116

出　　处：《福州大学学报（自然科学版）》2016年第5期644-648,共5页Journal of Fuzhou University(Natural Science Edition)

基　　金：国家自然科学基金资助项目(61471124)

摘　　要：针对传统的潜在狄利克雷分析(LDA)模型在提取评论主题时存在着计算时间长、计算效率低的问题,提出基于MapReduce架构的并行LAD模型建立方法.在文本预处理的基础上,得到文档-主题分布和主题-特征词分布,分别计算主题相似度和特征词权重,结合k-均值聚类算法,实现评论主题提取的并行化.通过Hadoop并行计算平台进行实验,结果表明,该方法在处理大规模文本时能获得接近线性的加速比,对主题模型的建立效果也有提高.Traditional latent Dirichlet analysis（ LDA） model in extracting thematic reviews exist long computing time and computing efficiency is low. Aiming at this problem,proposed MapReduce framework parallel lad model building method based on,in text preprocessing based,document-topic distribution and theme-feature word distribution,topic similarity and word feature weights were calculated,with k-means clustering algorithm,achieve comment on themes were extracted from the parallel.The experimental results show that the method can achieve near linear speedup in processing large scale text,and the effect of the model is improved by Hadoop parallel computing platform.

关 键 词：LAD模型 MAPREDUCE 评论主题 K-均值聚类算法 

分 类 号：TP391.1[自动化与计算机技术—计算机应用技术]