基于网页浏览行为的反爬虫研究  

Research on Anti-Spider Method Based on Web Browsing Behavior

作　　者：刘洋[1] LIU Yang(College of Computer Science, Sichuan University, Chengdu 610065)

机构地区：[1]四川大学计算机学院,成都610065

出　　处：《现代计算机》2019年第7期58-60,70,共4页Modern Computer

摘　　要：在大数据的背景下,数据的潜在价值被不断地挖掘出来。能够有效识别或阻挡爬取行为的反爬虫方法对于商业服务网站来说尤为重要。基于网页浏览行为,提出一种新的反爬虫方法。该方法通过对真实用户和网络爬虫浏览网页的行为进行特征提取,然后构造并使用决策树对一个用户是否属于爬虫进行预测。该方法对网络爬虫的敏感性高,并具有较低的假阴率。In the era of the big data, the potential value of data is constantly being explored. Anti-spider methods that effectively identify or block web spider crawling behavior are especially important for websites who offer commercial services. Proposes a new anti-spider method based on the behavior of browsing the web page. The method extracts feature from web browsing behavior of real users and web spider, and then constructs and uses decision tree to predict whether a user is web spider. This method has high sensitivity to web spider and a low false-negative rate.

关 键 词：网络爬虫 反爬虫方法 用户浏览行为 网站 

分 类 号：TP393.092[自动化与计算机技术—计算机应用技术] TP391.3[自动化与计算机技术—计算机科学与技术]