突发事件检测的MapReduce并行化实现  

Parallel Implementing Bursty Events Detection Using Map Reduce

作　　者：卓可秋 虞为[1] 苏新宁[1] 

机构地区：[1]南京大学信息管理学院,南京210023

出　　处：《现代图书情报技术》2015年第2期46-54,共9页New Technology of Library and Information Service

基　　金：国家社会科学基金项目"基于关联数据的图书馆语义云服务研究"(项目编号:12CTQ009);国家社会科学基金重大项目"面向突发事件应急决策的快速响应情报体系研究"(项目编号:13&ZD174);国家自然科学基金面上项目"面向知识服务的知识组织模式与应用研究"(项目编号:71273126);江苏省社会科学基金青年项目"基于语义云服务的数字阅读推广研究"(项目编号:14TQC003)的研究成果之一

摘　　要：【目的】在大数据环境下,从文本流中准确且快速地检测出特定领域的突发事件。【方法】利用Kleinberg突发检测方法和LDA主题模型方法,将其扩展到Map Reduce并行框架中,实现并行语料预处理、并行突发词检测、并行突发文档过滤和并行主题提取。【结果】对新闻文本流进行模拟仿真实验,结果表明,该并行方法在特定领域突发事件检测中准确率P、召回率R和调和平均值F分别最高可达87.50%、77.78%和82.35%。【局限】基于Map Reduce的并行方法难以实现大规模动态文本流在线(Online)实时(Real-time)突发事件检测。【结论】与传统串行突发事件检测方法相比,所构建的分布式并行化方法在保证检测结果正确性的同时,具有良好的可扩展性,性能得到较大提升。[Objective] In big data environment, this paper aims to accurately and quickly detect bursty events from the text stream. [Methods] Using Kleinberg bursty detection and LDA topic model, the method is extended to MapReduce framework to achieve parallel corpus predisposed, parallel detection of bursty word, parallel filtration of bursty document and parallel extraction of topic. [Results] The results of simulation experiments on the news text stream show that precision reaches 87.50%, recall reaches 77.78%, and F-measure reaches 82.35% with the parallel method to detect bursty events in specific areas. [Limitations] The MapReduce parallel method is difficult to achieve Online and Real-time detection ofbursty events with large-scale dynamic text stream. [Conclusions] Compared with the traditional serial detecting method of bursty events, the distributed parallel method not only guarantees the accuracy of detecting results, but also has a good scalability.

关 键 词：突发事件检测 MAPREDUCE 分布式处理 LDA 主题模型 

分 类 号：TP391.1[自动化与计算机技术—计算机应用技术]