DOLDA模型设计与主题演化分析  

Design of DOLDA model and analysis of theme evolution

作　　者：蒋权[1] 郑山红[1] 刘凯[1] 李万龙[1] JIANG Quan;ZHENG Shan-hong;LIU Kai;LI Wan-long(College of Computer Science and Engineering,Changchun University of Technology,Changchun 130012,China)

机构地区：[1]长春工业大学计算机科学与工程学院,吉林长春130012

出　　处：《计算机工程与设计》2018年第2期446-451,485,共7页Computer Engineering and Design

基　　金：吉林省自然科学基金项目(20130101060JC);吉林省教育厅"十二五"科学技术研究基金项目(2014131;2014125)

摘　　要：为解决OLDA模型挖掘大规模文档主题时计算效率低下和不能发现新主题的问题,提出一种分布式的DOLDA模型(distribute online LDA,DOLDA)。建立分布式矩阵存储主题-词项,设计一种动态负载均衡策略来提升计算速度和线程调度的性能,根据Zipf定律结合主题的遗传度提出一种文档权值设置方法。在Spark分布式计算平台的实验结果表明,相比OLDA模型,DOLDA模型能够提高近16%的加速比,有效地在线分析主题的演化。To solve the problem of lower computing efficiency and failing to detect new topics when mining themes of the massive set of documents in the OLDA,a distributed online LDA(DOLDA)model was presented.A distributed matrix was proposed to store the theme-word matrix,and a dynamic load balancing algorithm was adopted to improve the computing speed of the model and the performance of thread scheduling.Zipf’s law combined with the genetic degree of theme was introduced to set reasonable weights used in the documents.Through the Spark distributed computing platform,the results show that the model improves the speed-up ratio by nearly 16%in the process of dealing with large-scale topics of the texts,and effectively analyzes the evolution of online topics.

关 键 词：主题挖掘 分布式计算 在线的潜在狄利克雷分布模型 动态负载均衡 主题演化 

分 类 号：TP301.6[自动化与计算机技术—计算机系统结构]