中文搜索引擎分块倒排索引存储模式  

New inverted index storage scheme for Chinese search engine

作　　者：马健[1] 张太红[1,2] 陈燕红[1] 

机构地区：[1]新疆农业大学计算机与信息工程学院,乌鲁木齐830052 [2]中国农业大学信息与电气工程学院,北京100083

出　　处：《计算机应用》2013年第7期2031-2036,共6页journal of Computer Applications

基　　金：新疆维吾尔自治区科技攻关项目(200931103)

摘　　要：针对开源搜索引擎ASPSeek中的倒排索引的内部结构和访问模式,抽象出了倒排索引的形式化定义。为解决ASPSeek倒排索引更新困难和直接采用操作系统文件缓冲访问机制带来的效率问题,以125万张中文农业网页为样本,结合其特点,提出了一种块大小可变的分块倒排索引存储模式和基于CLOCK替代策略的专用缓冲管理机制。实验结果表明在禁用和启用缓冲的情况下,分块访问模式检索效率都优于ASPSeek。在启用缓冲并使用16万中文词条或5万高频中文词条作为测试集合的条件下,分块存储模式在访问倒排索引100万次后的检索时间趋于常量,即使以全部的827309个词条作为测试集合的条件下,分块存储模式在访问倒排索引200万次后的检索时间趋于收敛。After analyzing inverted index structure and access mode of an open source search engine-ASPSeek, this paper gave an abstract definition of ＂inverted index＂. In order to solve the difficulties of inverted index updating and the efficiency issues caused by directly accessing inverted index through file caching of operating system in ASPSeek, considering the characteristics of 1.25 million Chinese agricultural Web pages, this article proposed a new blocking inverted index storage scheme with a buffer mechanism which was based on CLOCK replacement algorithm. The experimental results show that the new scheme is more efficient than ASPSeek whether the buffer system is disabled or enabled. When the buffer system got enabled and 160 thousand Chinese terms or 50 thousand high-frequency Chinese terms were used as a test set, the retrieval time of new scheme tended to be a constant after one million accesses. Even when using entire 827309 terms as a test set, the retrieval time of new scheme began to converge after two million accesses.

关 键 词：倒排索引 搜索引擎 全文检索 分块结构 检索效率 

分 类 号：TP311[自动化与计算机技术—计算机软件与理论]