基于网站访问行为的匿名爬虫检测  

Anonymous Crawler Detection Based on Web Access

作　　者：邹建鑫 李红灵 

机构地区：[1]云南大学信息学院计算机科学与工程系,云南昆明650000

出　　处：《计算机技术与发展》2017年第12期103-107,114,共6页Computer Technology and Development

基　　金：国家自然科学基金资助项目(61562090)

摘　　要：通过分析和研究网络爬虫访问网页内容的行为,针对恶意网络爬虫伪装成浏览器访问网站难以甄别、网站日志检测工具不支持匿名网络爬虫检测等问题,总结了一些基于机器人排斥协议和基于爬虫行为的恶意网络爬虫检测算法。通过这些网络爬虫检测算法的启发,提出一种基于爬虫行为的检测匿名爬虫算法。该算法主要根据人为访问网站与网络爬虫访问网站时间的长短、访问的周期等,对网络爬虫进行检测,同时对算法进行了实验验证。实验数据来自一个服务器的网络日志。应用Python对实验数据进行处理,从而对网络匿名爬虫进行检测,并与当前主流的匿名网络爬虫检测算法进行比较。结果表明,该算法能够检测出并发量小的匿名的网络爬虫。By analysis and study of web crawler accessing web page, some detection algorithms of malicious web crawler are summarized based on robot exclusion protocol and crawling, aiming to the problem that it is difficult to identify website accessing from malicious web crawler disguised as a browser,and that web log detection tools don＇ t support anonymous web crawler detection. In consideration of a- bove algorithms, a new one to identify the camouflage web crawler is proposed based on crawling. It detects the web crawler mainly ac- cording to the length of access time and access cycle of website accessing form both human and crawler,and is verified by an experiment, the data of which is from a server web log. The experimental data are processed by Python for anonymous crawler detection. Compared with mainstream detection algorithm of anonymous web crawler,the proposed algorithm can detect the small amount of concurrent anony- mous web crawler.

关 键 词：网络爬虫 网络机器人排斥协议 网站访问行为 匿名爬虫检测 

分 类 号：TP393.08[自动化与计算机技术—计算机应用技术]