Web全文检索中间件的设计与应用  

Design and application of middleware for Web full-text retrieval

作　　者：张维刚[1] 徐永东[1] 雷小强[2] 何辉[1] 

机构地区：[1]哈尔滨工业大学(威海)计算机科学与技术学院,山东威海264209 [2]华中科技大学计算机科学与技术学院,武汉430074

出　　处：《计算机应用》2011年第8期2261-2264,共4页journal of Computer Applications

基　　金：国家自然科学基金资助项目(60803092)

摘　　要：为了更好地为Web站内检索提供服务,对Web全文检索的关键技术进行了研究,设计并实现了一个用于Web全文检索的中间件。该中间件通过多线程网页爬虫程序来抓取网页;对抓取结果采用基于Bloom-Filter的大规模URL去重算法来进行高效消重;应用所提出的新的基于标签的正文提取方法来进行网页正文提取,实验结果表明该正文提取方法是有效可行的;此外,为了改善用户的搜索体验,该中间件还提供了一些个性化搜索辅助功能。最后,实现了一个完整的博客搜索系统Boso(博搜)来验证中间件的性能,结果说明该中间件是可以投入实际应用的。To provide better Web search services,the key techniques of the full-text retrieval were studied and a middleware was designed and implemented.By using a multi-thread website crawler program,the Web pages of the given URLs were collected.Bloom-Filter algorithm was employed to get rid of large-scale duplicate URLs in the collected Web pages.A new content extraction approach based on the Web tags was presented to extract the full-text content of Web pages for indexing and searching.The experimental results verify the efficiency of the content extraction method.Furthermore,to improve the search experience of users,many personalized search assistances were provided by this middleware.Boso,a blog search engine,was developed to test and verify the presented middleware.The results show that the presented middleware can be applied to actual search engines.

关 键 词：Web全文检索 中间件 URL去重 网页标签 正文提取 

分 类 号：TP391.3[自动化与计算机技术—计算机应用技术]