面向跨境电商的网络抓取技术研究  

Research on web crawling technology for cross border e-commerce

作　　者：南博成 向杰益 沈晓红 李邱溢 潘君妮 张斌[1] Nan Bocheng;Xiang Jieyi;Shen Xiaohong;Li Qiuyi;Pan Junni;Zhang Bin(Zhejiang Yuexiu University of Foreign Language,Shaoxing,Zhejiang 312000,China)

机构地区：[1]浙江越秀外国语学院电子商务系,浙江绍兴312000

出　　处：《计算机时代》2018年第8期13-16,共4页Computer Era

基　　金：2017年国家级创业创新训练项目<面向跨境电商的网络抓取技术应用研究>(201712792015);2016年度浙江越秀外国语学院国际商学院重点课题

摘　　要：网络爬虫已成为提高跨境电商数据化运营效率的一项热门技术。本文通过叙述商业爬虫软件采集商品页面信息的基本方法,论述了爬虫程序提取商品价格与评价信息的常用算法,分析了抓取Ajax生成的Json信息、实现动态翻页和应对网站的反对爬虫的技术和方法。实践证明,由于电商平台反爬虫策略会动态调整,如何利用人工智能技术更好地模仿人类行为来实现有效的信息采集是跨境电商爬虫技术提升的要点。Web crawler is a popular technology to promote the efficient operations of cross border e-commerce in big data era.This paper analyzes the normal method of crawling data with commercial crawler software,discusses the usual algorithms of extracting goods evaluations and prices with web crawler program,and describes the method of extracting information of Json file created by Ajax,the method of turning page and the method of reducing the chance that would be detected by anti-crawler techniques and methods of web sites.The practice has proved that the effective way of extracting information for cross border e-commerce is to change the web crawler's access method,and let it imitates human behavior by using AI technology.

关 键 词：跨境电商 爬虫 AJAX JSON 反爬虫 

分 类 号：TP391.1[自动化与计算机技术—计算机应用技术]