Python爬虫技术在学术聚合系统中的应用  

The Application of Python Crawler Technology in Academic Aggregation Systems

作　　者：崔梦银 邓茵 刘满意 CUI Mengyin;DENG Yin;LIU Manyi(Guangdong University of Science and Technology,Dongguan 523083,China;Shenzhen Huanxuntong Technology Co.,Ltd.,Shenzhen 518000,China)

机构地区：[1]广东科技学院,广东东莞523083 [2]深圳市环讯通科技有限公司,广东深圳518000

出　　处：《现代信息科技》2024年第10期68-74,共7页Modern Information Technology

基　　金：广东科技学院校级科研项目(自然科学类)一般项目(GKY-2023KYYBK-15);广东科技学院大学生创新创业训练计划项目(202313719004)。

摘　　要：爬虫技术是搜索引擎和信息网站获取数据的核心技术之一,专用的网络爬虫能够在短时间内从网络上抓取大量有用数据。基于为研究者提供所需学术资源的目的,研究了爬虫技术在爬取学术网站论文数据中的应用。分析了Python爬虫技术在学术聚合系统中的应用,借助大数据技术手段对所爬取的学术数据进行存储、清洗、聚合、消歧和融合。Python爬虫技术在学术聚合系统中起着关键作用,助力研发人员构建强大的数据聚合和分析平台,为学术研究人员提供有价值的信息资源,对学术研究、文献检索和信息发现都具有重要意义。Crawler technology is one of the core technologies for search engines and information websites to obtain data.Specialized web crawlers can quickly crawl a large amount of useful data from the network.In order to meet the needs of researchers crawling academic paper data on academic websites to obtain academic resources,the application of Python crawler technology in academic aggregation systems is studied.With the help of big data technology,the crawled academic data is stored,cleaned,aggregated,disambiguated,and fused.Python crawler technology plays a crucial role in academic aggregation systems,helping developers build powerful data aggregation and analysis platforms,providing valuable information resources for academic researchers,and is of great significance for academic research,literature retrieval,and information discovery.

关 键 词：Python爬虫 学术资源 大数据技术 学术聚合系统 

分 类 号：TP311.5[自动化与计算机技术—计算机软件与理论] TP312[自动化与计算机技术—计算机科学与技术]