基于数据挖掘的网络信息采集系统设计  

Design of Network Information Collection System Based on Data Mining

作　　者：王军[1] WANG Jun(Suqian higher normal school suqian,Suqian Jiangsu 223800,China)

机构地区：[1]宿迁高等师范学校,江苏宿迁223800

出　　处：《信息与电脑》2022年第22期160-162,共3页Information & Computer

摘　　要：网络信息采集的效率直接影响着用户查询信息的时新性,为了满足人们海量的数据需求,研究基于数据挖掘的网络信息采集系统。在系统总体结构设计中,基于hadoop分布式集群,形成中心化拓扑主从结构,负责网络信息的采集任务。获取网页源代码,计算网页更新概率,得到大规模平行网页信息;基于数据挖掘寻找信息关联关系,过滤无用信息;建立网络信息采集模块、变更管理模块和网站登录模块,利用爬取技术完成网络信息采集,实现系统设计。经实验论证分析,设计系统信息采集较快。The efficiency of network information collection directly affects the timeliness of user query information.In order to meet people’s massive data needs,a network information collection system based on data mining is studied.In the overall structure design of the system,a centralized topology master-slave structure is formed based on the Hadoop distributed cluster,which is responsible for the collection of network information.Obtain web page source code,calculate web page update probability,and obtain large-scale parallel web page information;search for information correlation based on data mining,filter useless information;establish network information collection module,change management module and website login module,and use crawling technology to complete network information Collect,realize the system design.Through experimental demonstration and analysis,the design system information collection is faster.

关 键 词：数据挖掘 爬取技术 信息采集 

分 类 号：TP311.13[自动化与计算机技术—计算机软件与理论]