基于网络爬虫的网页目标数据精准抓取方法  

Accurate capturing method of web target data based on web crawler

作　　者：高祖彦[1] GAO Zuyan(Dean’s Office,Enshi Polytechnic,Enshi,Hubei 445000,China)

机构地区：[1]恩施职业技术学院,湖北恩施445000

出　　处：《自动化与仪器仪表》2024年第10期38-42,共5页Automation & Instrumentation

基　　金：恩施职业技术学院自然科学研究项目:“基于Python的网络爬虫技术研究”(EZYQNZK201906);恩施州科技局项目:旅游景区游客导览智能分析系统及景区智慧管控制度与应急指挥平台建设研究(D20220012)。

摘　　要：从海量的网络数据中提取出有价值的信息,为数据分析等应用提供强大的数据支持,网页目标数据的精准抓取的准确性和效率直接影响到后续数据分析和处理的效果,目前方法无法保障爬取数据的数量和质量,导致抓取到的数据不够完善,容易出现重复性数据。因此,提出基于网络爬虫的网页目标数据精准抓取方法。详细分析网络爬虫的爬取过程,采用主题爬虫策略中的爬行器爬行指定网页,采用链接分析器中的基于HITS算法的网页链接分析算法,计算所有链接的权威值,选择权威值较高的链接作为下载目标;再利用目标数据主体识别器中的向量空间模型计算网页中待抓取数据与主题特征向量之间的相关性,按照相关性大小依次抓取网页中的数据,直至达到设定的停止条件后,从下载的网页中精准抓取所需数据。经实验分析可知:该方法可精准评估不同网页的权威性,在抓取数据时,可以保持较高的目标数据收获率,还能够提供一定的实时性与灵活性。Extracting valuable information from massive network data provides powerful data support for data analysis and other applications.The accuracy and efficiency of accurate crawling of web page target data directly affect the effect of subsequent data analysis and processing.The current methods cannot guarantee the quantity and quality of crawled data,which leads to the incompleteness of the crawled data and the easy occurrence of repetitive data.In order to propose an accurate method to capture web page target data based on web crawler.Detailed analysis of the crawling process of web crawlers,using the crawler in the theme crawler strategy to crawl specified web pages,using the web link analysis algorithm based on HITS algorithm in the link analyzer,calculating the authoritative values of all links,and selecting links with higher authoritative values as download targets;Reuse the vector space model in the target data subject recognizer to calculate the correlation between the data to be crawled and the topic feature vector in the webpage.Grab the data in the webpage in order of correlation until the set stop condition is reached,and then accurately grab the required data from the downloaded webpage.Through experimental analysis,it can be concluded that this method can accurately evaluate the authority of different web pages,maintain a high target data harvesting rate when crawling data,and provide a certain degree of real-time and flexibility.

关 键 词：网络爬虫 网页目标数据 精准抓取 主题爬虫 向量空间模型 收获率 

分 类 号：TP391[自动化与计算机技术—计算机应用技术]