Web语料抓取中基于相似度的URL过滤规则生成算法  

A URL Filtering Generation Algorithm Based on Similarity Degree for Web Crawling

作　　者：陈荟慧[1] 舒云星[1] 林丽[2] 

机构地区：[1]洛阳理工学院计算机与信息工程系,洛阳471023 [2]解放军外国语学院亚非语系,洛阳471003

出　　处：《模式识别与人工智能》2014年第7期631-637,共7页Pattern Recognition and Artificial Intelligence

基　　金：教育部哲学社会科学研究重大课题攻关项目(No.12JZD014)资助

摘　　要：Web语料是语料库的重要组成部分,但对冗余URL的访问开支影响大规模语料爬取工作的质量和效率,使用高效的URL过滤规则可提高Web爬取的质量和效率.因网站虚拟目录下的文件分布不均匀,为发现目标文件聚集区域,提出一种生成URL过滤规则的方法.该方法使用正则表达式将URL元素通配化,归并相同元素后划分为子集,再计算子集内URL之间的相似度,并根据相似程度较高的URL构造虚拟目录树,基于虚拟目录树生成语料爬取的URL过滤规则和分类规则.文中详细介绍虚拟目录树的生成算法,并通过实验对比不同相似度阈值对目录树生成结果和URL过滤效果的影响.Web text is an important component of the corpus, however, unnecessary time consumption for visiting redundant URLs influences the quality and efficiency of the large scale web crawling. The quality and efficiency of Web crawling can be promoted by using high effective URL filtering rules. The distribution of files in the virtual directories of a website is uneven and a URL filtering rule generation method is introduced to discover the c!ustering region of target files. Firstly, URLs are transformed into regular expressions and they are divided i/ato many groups by clustering same regular expressions. Then, the similarity degrees between URLs in one group are calculated and the virtual path tree is constructed by using URLs with higher similarity degrees. Finally, the virtual path tree is utilized to generate URL filtering rules and classification rules for Web crawling. The algorithms for generating virtual path tree are introduced in detail and the experimental results of the generated virtual path trees and the filtered URLs are compared by using different similarity degree thresholds.

关 键 词：URL相似度 Web语料爬取 URL过滤 语料分类 

分 类 号：TP391.1[自动化与计算机技术—计算机应用技术]