CCDet:一种高效的大规模中文重复网页检测方法  

CCDet:An Efficient Detection Method for Large-Scale Duplicate Chinese Web Pages

作　　者：韦永壮[1] 袁春风[1] 黄宜华[1] 

机构地区：[1]南京大学计算机科学与技术系,南京210046

出　　处：《计算机研究与发展》2013年第S2期140-152,共13页Journal of Computer Research and Development

基　　金：国家自然科学基金项目(61072152);江苏省工业支撑计划基金项目(BE2011172)

摘　　要：重复文档检测是信息检索领域中一个非常重要的问题.由于网页结构和内容的复杂性,现有方法在网页查重上没有达到很好的准确性,且只有少量工作用于处理包含关系网页检测问题;同时,由于网页数量的巨大,重复网页检测处理时需要考虑大规模数据的并行化算法.提出一种基于句号特征的大规模重复中文网页检测方法CCDet.CCDet采用了一种基于中文句号特征来完成重复文档的相似性比对方法,与现有的主要重复网页检测算法相比,CCDet大幅提高了检测具有重复关系网页和具有包含关系网页的准确性,并拥有较高的检测效率.同时,为了适应大规模新闻网页的查重处理,使用MapReduce编程框架实现了并行化的CCDet算法,使之能够并行化地进行重复网页检测.实验结果表明,并行化的CCDet算法具有较好的检测效果和计算性能,并具有良好的可扩展性.Duplicate document detection is an important topic in information retrieval domain.For the complexity in Web pages structure and content,current duplicate detection algorithms cannot achieve good accuracy and few algorithms refer to the containment detection of Web pages.In the mean time, for the reason that duplicate Web page detection needs to process large-scale datasets,we have to consider a parallel algorithm to deal with large-scale Web page duplicate detection.This paper presents a large-scale Chinese Web page duplicate detection algorithm,called CCDet.Based on Chinese period feature to compute the similarity of Web documents,CCDet can highly improve the accuracy and efficiency of duplicate and containment detection.Furthermore,for dealing with large scale of news Web page duplicate detection,we also apply MapReduce program framework to implement CCDet algorithm to parallelized detect duplicate Web pages.Experimental results show that the parallelized CCDet algorithm achieves good performance and scalability.

关 键 词：CCDet算法 重复网页检测 中文句号特征 索引剪切 

分 类 号：TP393.092[自动化与计算机技术—计算机应用技术] TP391.3[自动化与计算机技术—计算机科学与技术]