面向BitTorrent种子文件获取的网络爬虫技术研究  

Research on BitTorrent Seed File Oriented Web Crawling

作　　者：苏马婧[1] 叶麟[1] 史建焘[1] 

机构地区：[1]哈尔滨工业大学计算机科学与技术学院,哈尔滨150001

出　　处：《智能计算机与应用》2013年第3期7-13,共7页Intelligent Computer and Applications

基　　金：973计划资助项目(2007CB311101);863计划资助项目(2009AA012437)

摘　　要：分析了当前主题爬虫在面向BT种子文件获取应用上存在的问题,从提高BT种子获取速率、提高覆盖率和降低种子获取延时的角度出发,提出了基于Hash的去重机制,给出了爬虫自动登录的实现方法,设计了AJAX页面解析引擎,提出批量抓取和增量抓取相结合的数据抓取机制、历史数据和更新数据相结合的数据存储机制。通过设计并实现一个基于爬虫方式的BT种子文件获取系统证明了这几种方法能使系统整体性能平均提高30%～50%。这些技术和方法也可应用于其他主题爬虫。This article analyzes some difficulties in BitTorrent seed file crawling and explores several approaches of establishing efficient torrent crawler.Some schemes are proposed to improve torrents coverage rate,including a highly efficient duplicated files and URLs deletion method based on hash,automatic login agent,Ajax parsing engine.Different crawling policies and data storage policies for history torrents and new torrents are used to reduce the delay of fetching newly-added torrents.Experimental results indicate that these methods can obtain high freshness and coverage rate.These technologies can also be used in general-crawlers and other topic-focused crawlers.

关 键 词：BITTORRENT BT种子爬虫 去重机制 自动登录 AJAX解析 

分 类 号：TP393[自动化与计算机技术—计算机应用技术]