网页去重策略  

The Strategy on Processing Replicated Web Collections

作　　者：高凯[1] 王永成[1] 肖君 

机构地区：[1]上海交通大学计算机科学与工程系,上海200030 [2]上海远程教育集团,上海200086

出　　处：《上海交通大学学报》2006年第5期775-777,782,共4页Journal of Shanghai Jiaotong University

基　　金：国家高技术研究发展计划(863)项目(2002AA119050)

摘　　要：提出基于同源网页去重与内容去重的策略.通过对网址URL进行哈希散列完成对同源网页的去重,并对内容相同或近似的网页采用基于主题概念的去重判断.实验表明,该方法有效且去重效果良好.基于上述算法实现了教育资源库教育资讯搜索引擎系统.This paper presented techniques on how to build an effective crawler to collect non-replicative Web pages. A novel Hash function was proposed, together with a content-oriented approach, to filter based on URLs and contents. On one hand, this technique can parallelize crawling process while minimize the overlap effectively. On the other hand, it can identify those near-duplicated collections. The experimental results show the feasibility of the approach. On the basis of this work, the implementation of an educational search engine was presented in the end.

关 键 词：信息检索 搜索引擎 哈希函数 主题概念 

分 类 号：TP393[自动化与计算机技术—计算机应用技术]