移动新闻自适应采集方法研究  

Research on adaptive crawling method of mobile news

作　　者：胡建洪[1] 徐建[1] 董克源 高鑫 Hu Jianhong;Xu Jian;Dong Keyuan;Gao Xin(School of Computer Science&Engineering,Nanjing University of Science&Technology,Nanjing 210094,China)

机构地区：[1]南京理工大学计算机科学与工程学院,南京210094

出　　处：《计算机应用研究》2018年第9期2798-2801,共4页Application Research of Computers

基　　金：赛尔网络下一代互联网技术创新项目(NGII20151202)

摘　　要：通过分析面向传统新闻门户网站和面向移动新闻应用的新闻采集方法的异同,提出了一种面向移动新闻应用的自适应新闻采集方法。该方法先通过代理对移动新闻应用与服务器通信的数据包进行解析,构造出移动新闻应用的站点地图;然后通过模拟用户行为自适应地增量采集新闻数据,提高更新效率并保证新闻采集的完整性。以国内外20多家主流的移动新闻应用为实验对象开展验证,实验结果表明该方法的有效性和实时性。By analyzing the similarities and differences between the news gathering method for traditional news portal and the news gathering method for mobile news application,this paper proposed an adaptive news gathering method for mobile news application.The method firstly analyzed the data packet of the mobile news application and the news server through the agent,constructed the site map of the mobile news application,and then incremental gathered the news data adaptively based on simu-lating the user behavior,ensured the integrity of news gathering and improved the efficiency of updating.Experiments using more than 20 domestic and foreign mainstream mobile news applications show that the method is effective and real-time.

关 键 词：移动新闻应用 新闻爬虫 站点地图 自适应 增量更新 

分 类 号：TP391.1[自动化与计算机技术—计算机应用技术]