基于Python多重解析的图像爬虫的设计与实现  

Design and Implementation of Images Crawler Based on Python and Multiple Analysis

作　　者：杨松 刘佳欣 

机构地区：[1]大连外国语大学软件学院网络空间多语言大数据智能分析研究中心,辽宁大连116044

出　　处：《工业控制计算机》2021年第2期99-101,104,共4页Industrial Control Computer

基　　金：辽宁省社科规划基金(L18BTQ005);辽宁省教育厅科学研究项目(2019JYT07)。

摘　　要：深度学习已经被广泛应用在自然语言处理和图像识别领域。深度学习模型需要大量的训练数据,而大量的训练数据能够有效地防止模型的过拟合。图像情感分类研究基于社交媒体的海量图像集,这些数据集往往不直接提供图像下载,而采用记录图像地址信息的专有格式文件。这些文件通常存在非法字符、解析困难、图像真实地址隐藏等问题,这给图像地址解析和下载带来诸多困难。以Python为工具,研究多重解析技术解决非法字符、大容量文件标签遍历和解析图像真实地址问题,并利用异常处理和多线技术实现图像准确快速地下载,帮助研究人员方便地采集图像数据集。Deep learning has been widely applied in natural languages processing and images recognition fields.Deep learning model needs lots of training data,which are effective to prevent overfitting for model.The researches on images sentiment classification are based on massive images datasets of social media,and these datasets don’t provide images downloads directly,but give mass files which record images address information in special format.These files have usually some questions,such as illegal characters,parsing difficulties and hiding real images address and so on,which brings many difficulties to parse files and download images.This paper uses Python as tool to research multiple analysis technique,which deal with illegal characters,visit labels of mass file,and parse real images addresses,and use exception processing and multi-threading techniques to downloading images accurately and quickly.It can help researchers collect images datasets conveniently.

关 键 词：多重解析 非法字符过滤 大容量文件 地址解析 图像爬虫 多线程 

分 类 号：TP391.41[自动化与计算机技术—计算机应用技术] TP18[自动化与计算机技术—计算机科学与技术]