基于大数据新闻网站文本挖掘的网络舆情监测设计与实现  

Design and Implementation of Network Public Opinion Monitoring Based on Big Data News Website Text Mining

作　　者：梁永春 焦文强 田立勤 LIANG Yongchun;JIAO Wenqiang;TIAN Liqin(College of Computer,North China Institute of Science and Technology,Yanjiao,065201,China)

机构地区：[1]华北科技学院计算机学院,北京东燕郊065201

出　　处：《华北科技学院学报》2018年第4期82-87,92,共7页Journal of North China Institute of Science and Technology

基　　金：国家自然科学基金项目(61163050)

摘　　要：当今中国,网民人数已经超过人口总数的一半,因此网络舆情监测具有十分重要的意义。本文首先应用网络爬虫技术实现对新闻报道和对应网民评论文本数据获取。因为数据量大,所以选择Hadoop集群进行文本数据储存。其次,通过中文分词技术对文本数据按词语进行拆分,并对得到的词语进行过滤和挑选,得到关键词。新闻报道文本中获取的关键词用于新闻类型与主题的判别,网民评论中的关键字反映了网民对此新闻报道的观点和态度。最后,应用此方法,对"中美贸易战"事件进行网络舆情监测,从获得的新闻报道主题和网民评论的关键字表明,本文介绍的网络舆情监测方法具有可行性和实用性。Nowadays,the number of netizens in China has exceeded half of the total population,so it is very important to monitor the public opinion on the Internet. In this paper,firstly,web scraping with Python is applied to achieve news report and corresponding user comments data. Because of the big data,Hadoop are selected to store the data. Secondly,Chinese word segmentation technology is used to separate the data according to Chinese words,and the words obtained are filtered and selected to get the key words. The keywords obtained from news reports are used to distinguish news types and topics. The keywords in the comments reflect the opinions and attitudes of the netizens about the reports. Finally,this method is applied to monitor the online public opinion monitoring of the China-U. S. trade frictions. The topics of news reports and the keywords of comments from netizens show that the online public opinion monitoring method introduced in this paper is feasible and practical.

关 键 词：舆情监测 网络爬虫 中文分词 

分 类 号：TP391[自动化与计算机技术—计算机应用技术]