基于证据增强和局部语义交互的多模态虚假新闻检测  

Multimodal Fake News Detection Based on Evidence Enhancement and Local Semantic Interaction

作　　者：钟将[1] 高晋鹏 黄敬旺 杨钰铭 ZHONG Jiang;GAO Jin-Peng;HUANG Jing-Wang;YANG Yu-Ming(College of Computer Science,Chongqing University,Chongqing 401331)

机构地区：[1]重庆大学计算机学院,重庆401331

出　　处：《计算机学报》2025年第3期556-571,共16页Chinese Journal of Computers

基　　金：国家自然科学基金(62176029);重庆市科技创新与应用发展专项基金(CSTB2023TIAD-KPX0064、CSTB2022TIAD-KPX0206)的部分资助。

摘　　要：多模态虚假新闻检测的目标是判断新闻中图像和文本内容的真实性。现有虚假新闻检测方法主要存在以下两种问题:(1)现有方法通常从整体语义角度融合图文特征,忽略了图文局部语义之间的联系,导致模型不能有效捕捉图文局部语义差异性;(2)新闻的真实性往往基于可靠的证据和事实,现有方法仅依赖新闻本身的图像和文本难以判断其真假。鉴于此,本研究提出了一种基于证据增强和局部语义交互的多模态虚假新闻检测模型。针对新闻缺乏事实依据的问题,该模型引入证据文本并设计了一种证据增强方法,该方法通过证据文本筛选网络,剔除证据文本中的冗余信息,并利用自注意力模块实现新闻文本的证据增强。同时,为了增强图像语义信息,该模型先从图像块中提取局部特征,再通过双向GRU图像语义增强网络,捕获图像序列特征的上下文关系,并利用自注意力模块将图像中嵌入的文字作为新闻背景信息融入图像特征。最后,针对局部语义信息交互问题,该模型使用交叉注意力模块,学习证据增强后的文本特征和语义增强后的图像特征之间的互补信息,增强细粒度的局部语义交互,实现多模态虚假新闻的精确检测。在Weibo数据集与MR2中英文数据集上的实验结果表明,本文提出的模型性能优于基线方法,在各数据集的准确率上分别提高了0.8%、2.4%、4.9%。此外,在IKCEST第五届“一带一路”国际大数据竞赛中,使用该模型指定的方案从全球3809个方案中取得第一的成绩,证实了该方案的有效性。With the continuous development of information technology and the widespread popularity of social media in recent years,a large amount of multimodal information is generated on the Internet every day,among which false news is widely exposed and spread through social networks.Effective false news detection methods can reduce the harm caused by false news to society.Current multimodal false news detection methods can obtain more prior semantic knowledge through pre-training models and use the overall semantics between images and texts to guide model decisions.Although these methods can detect false news with large semantic differences between images and texts,they cannot distinguish the local semantic differences between images and texts well.In addition,shallow overall semantic fusion cannot fully tap the prior semantic knowledge of each modality,and deep local semantic interaction is required to effectively capture the semantic differences between different modalities.At the same time,news usually focuses on what is happening at the moment.Without relevant evidence content,the authenticity of news reports cannot be judged.In fact,external evidence can provide different perspectives and viewpoints to the model to assist the model in judging the credibility of news.Inspired by this,this paper proposes a cross-modal deep local semantic interaction multimodal fake news detection model based on additional evidence.In response to the problem that news lacks factual basis,the model introduces evidence text and designs an evidence enhancement method.The introduction of evidence text verifies the authenticity of news content in multiple ways.In response to the problem of local semantic information interaction,the fine-grained interaction of local semantics of images and texts is achieved through the cross-attention mechanism,which improves the effectiveness and rationality of fake news detection.Specifically,the model first uses a multimodal feature extraction network to represent the semantic information of image data an

关 键 词：多模态虚假新闻检测 证据增强 局部语义交互 证据文本筛选 图像语义增强 

分 类 号：TP391[自动化与计算机技术—计算机应用技术]