面向大规模数据的在线新事件检测  

Online New Event Detection for Large-scale Data

作　　者：蔡偃武 高大启[1] 阮彤[1] 蒋锐权[2] 

机构地区：[1]华东理工大学计算机科学与工程系,上海200237 [2]上海证券交易所技术开发部,上海200120

出　　处：《计算机工程》2014年第10期37-42,共6页Computer Engineering

基　　金：国家科技支撑计划基金资助项目"证券业云平台研发与运营"(2012BAH13F02)

摘　　要：通过分析基于新闻要素的在线新事件检测算法的时间消耗,提出一种面向大规模数据环境的在线新事件检测算法。该算法利用基于倒排索引的高效相似报道搜索机制,有效减少单路径聚类算法中的相似度比较次数。通过对报道预处理、报道与事件比较以及索引搜索这3个过程的并行化,提高算法在多机环境下的运行效率和可伸缩性。实验结果表明,该算法在不影响漏检率和误检率的基础上,提高了新事件检测的速度,并且在千万到亿级别的报道规模下,其吞吐量达到150条/s^200条/s。Through analyzing the time consumption of the existing online New Event Detection（ NED） algorithm based on news elements, this paper improves an online NED algorithm for large-scale data environment. The algorithm uses efficient reported similar search mechanism based on inverted index to reduce the similarity comparison of single path clustering algorithms. Through parallelization of report pretreatment, report and event comparison, index search, it improves the efficiency and scalability of the algorithm in multimachine. Experimental result shows that the algorithm can greatly improve new event detection speed without affecting the miss probability and false-alarm probability, and its throughput reaches 150~200 reports/s at the scale of 10~100 million reports.

关 键 词：新事件检测 单路径聚类 大规模数据 并行计算 倒排索引 MapReduce架构 

分 类 号：TP391[自动化与计算机技术—计算机应用技术]