网页去重方法研究及算法设计  

Study on duplicated removal method Web pages and algorithm design

作　　者：张素智[1] 樊得强[1] 

机构地区：[1]郑州轻工业学院计算机与通信工程学院,河南郑州450002

出　　处：《郑州轻工业学院学报（自然科学版）》2010年第2期63-66,74,共5页Journal of Zhengzhou University of Light Industry:Natural Science

基　　金：河南省重点科技攻关项目(082102210054);河南省自然科学基金资助(0411010500)

摘　　要：针对在网页检索结果中经常出现内容相同的冗余页面的问题,为了按用户偏好对搜索结果进行筛选,提出了一种通过中文字主题要素学习网页内容的网页去重算法.该算法通过抽取网页要素中用户偏好的短语来抽取网页的内容,根据学习的网页内容计算其相似度来判断网页的重复度.经与相似方法比较,结果表明,该方法能够完成针对中文内容网页的去重,具有较高的查全率和查准率.In view of the Web page retrieval result, the user frequently obtains the same content redundant page. In order to solve this problem, one kind of duplicated Web pages removal algorithm according to the user＇s interest is proposed to study Web pages content on elements of Chinese topic. This method follows the three steps ： First, extract phrase of the user＇s interest which is Chinese elements content of Web pages, then study Web pages content extracted and finally, calculates their similarity according to the study news content to judge the heavy multiplicity of the news homepage. Compared with simliar methods, the experimental result indicated that, in view of the Chinese content web pages duplicated, this method obtains the high recall and the accuracy ratio.

关 键 词：中文字主题要素 模糊匹配 去重算法 查全率 差准率 

分 类 号：TM912.2[电气工程—电力电子与电力传动]