基于局部敏感哈希算法的文档内容相似性判断算法  

Document content similarity judgment algorithm based on local sensitive Hash algorithm

作　　者：曹雨 李刚 金义 马路遥 宋永春 刘凯 CAO Yu;LI Gang;JIN Yi;MA Luyao;SONG Yongchun;LIU Kai(Material Branch of State Grid Anhui Electric Power Co.,Ltd.,Hefei 230061,China;Anhui Jiyuan Software Co.,Ltd.,Hefei 230031,China)

机构地区：[1]国网安徽省电力有限公司物资分公司,安徽合肥230061 [2]安徽继远软件有限公司,安徽合肥230031

出　　处：《电子设计工程》2024年第18期187-190,195,共5页Electronic Design Engineering

基　　金：国网安徽电力2019年科技项目(5212T01900KJ)。

摘　　要：当前普遍采用全局逐一度量的方式进行文档内容相似性判断,工作量十分巨大,因此研究一种基于局部敏感哈希算法的文档内容相似性判断算法。针对文档实施中文分词、去停用词处理。筛选中心关键词并以此为基础将文档表示成空间向量的形式。将语料库中的文档与待检测文档都哈希到哈希桶中,在哈希桶中度量待检测文档与语料库文档之间的相似性,并通过阈值实现文档内容相似程度的判断。实验结果表明,采用所研究算法可以实现对文档内容相似性度的计算并得出了对应的相似程度。Currently,a global and individual measurement method is commonly used for document content similarity judgment,with a huge workload.Therefore,a document content similarity judgment algorithm based on local sensitive Hash algorithm is studied.Implement Chinese word segmentation and word removal for documents.Filter the central keywords and represent the document as a spatial vector based on this.Hash both the document in the corpus and the document to be detected into a Hash bucket,measure the similarity between the document to be detected and the corpus document in the Hash bucket,and determine the similarity of the document content through a threshold.The experimental results show that using the algorithm studied can achieve the calculation of document content similarity and obtain the corresponding similarity level.

关 键 词：局部敏感哈希算法 预处理 关键词筛选 内容相似性 空间向量 

分 类 号：TN99[电子电信—信号与信息处理]