基于前缀剪枝的大规模向量空间相似检索框架  

A Large-Scale Vector Space Similarity Retrieval Framework Based on Prefix Pruning

作　　者：刘健博 邓凌风 李文海[2] 田野 LIU Jianbo;DENG Lingfeng;LI Wenhai;TIAN Ye(Wuhan DNect Technology ltd.,Wuhan 430205,China;School of Computer Science,Wuhan University,Wuhan 430072,China;School of Software Engineering,Hubei Open University,Wuhan 430074,China)

机构地区：[1]武汉数博科技有限责任公司,湖北武汉430205 [2]武汉大学计算机学院,湖北武汉430072 [3]湖北开放大学软件工程学院,湖北武汉430074

出　　处：《软件导刊》2024年第6期92-97,共6页Software Guide

基　　金：武汉市重点研发计划项目(2023010402040006)。

摘　　要：针对大规模文本集合下基于权重的相似性查询问题,提出一种支持前缀剪枝的高效检索框架。首先给出向量空间模型下相似性及其带权前缀定义,理论证明了带权前缀剪枝的正确性;其次,面向大规模文本查询,提出一种新的倒排索引结构,利用索引叶节点维护记录的前缀权重,并基于该索引构建高效的相似检索算法;最后,在TF/IDF权重策略下证明该方法能够有效支持大规模带权相似检索。结果表明,其查询效率较Lucene的归并验证策略提升了5倍以上。Aiming at the problem of weight-based similarity query under large-scale text collection,an efficient retrieval framework supporting prefix pruning is proposed.Firstly,we give the definition of similarity and its weighted prefix under the vector space model,and theoretically prove the correctness of weighted prefix pruning;then,for large-scale text query,we propose a new inverted index structure,use the index leaf nodes to maintain the prefix weights of the records,and construct efficient similarity retrieval algorithms based on the index;finally,we prove that the meth-od can effectively support large-scale similar retrieval with weights,and the results show that its query efficiency is more than 5 times higher than that of Lucene's subsumption verification strategy.

关 键 词：前缀剪枝 TF/IDF 向量空间 倒排索引 信息检索 数据库 

分 类 号：TP311.13[自动化与计算机技术—计算机软件与理论]