双分支线索深度感知与自适应协同优化的多模态虚假新闻检测  

Multimodal Fake News Detection via Two-Branch Deep Clue Perception and Adaptive Collaborative Optimization

作　　者：钟善男 彭淑娟[1,3] 柳欣[1,2] 王楠楠 李太豪[2] ZHONG Shan-Nan;PENG Shu-Juan;LIU Xin;WANG Nan-Nan;LI Tai-Hao(College of Computer Science and Technology,Huaqiao University,Xiamen,Fujian 361021;Zhejiang Lab,Hangzhou,311121;Fujian Key Laboratory of Big Data Intelligence and Security,Huaqiao University,Xiamen,Fujian 361021;State Key Laboratory of Integrated Services Networks,Xidian University,Xi’an 710126)

机构地区：[1]华侨大学计算机科学与技术学院,福建厦门361021 [2]之江实验室,杭州311121 [3]华侨大学福建省大数据智能与安全重点实验室,福建厦门361021 [4]西安电子科技大学空天地一体化综合业务网全国重点实验室,西安710126

出　　处：《计算机学报》2023年第12期2612-2625,共14页Chinese Journal of Computers

基　　金：之江实验室开放课题(2021KH0AB01);国家自然科学基金联合基金重点项目(U22A2096);福建省自然科学基金项目(2020J01083,2020J01084)资助。

摘　　要：深度学习方法促使多模态虚假新闻检测领域快速发展,现有的检测模型通常从全局角度学习新闻图文间的跨模态语义关联,并利用共享语义内容获取检测的关键信息.然而,新闻内部的局部语义差异可能会限制模型有效利用跨模态语义关联的能力,其中潜在的非共享语义内容作为重要线索能够有效揭示虚假新闻的篡改意图和目的.为了解决上述问题,本文提出了一种双分支线索深度感知与自适应协同优化的多模态虚假新闻检测模型.该模型首先从图像显著区域和文本语义单词中提取细粒度的新闻特征,并使用跨模态加权残差网络从中学习共享语义线索.同时,根据所有图像区域和文本单词之间的语义相关性,双分支图文线索感知模块显式地建模共享与非共享语义内容的语义关联.其中,线索关联优化分支对两类语义内容的关联边界持续迭代优化,促使模型准确区分非共享语义线索;线索关联分析分支刻画两类语义内容的可信程度,并在此基础上引导模型实现线索的自主融合.通过上述自适应协同优化框架,本文提出的模型能够在复杂新闻语境下进行线索的深度感知与融合,实现更准确、更可解释的多模态虚假新闻检测.在广泛使用的中英文真实数据集上的实验结果表明,本文提出的模型明显优于基线方法,在准确率和虚假新闻检测精确率上分别平均提高了4.85%和4.50%.Deep learning methods are able to learn the high-level semantic features and significantly promote multimodal fake news detection performances.In the literature,existing multimodal fake news detection models usually learn the cross-modal semantic correlations across news image and text from a global perspective,and utilize their shared information to infer the key clues for detection.Although such approaches are able to detect the obvious multimodal fake news,these global modeling methods cannot well differentiate the local semantic differences within the news and therefore may degrade the detection performance.Indeed,the unshared semantic content serves as an important clue that can directly reveal their tampering intentions and purposes.Inspired by these findings,this paper proposes a multimodal fake news detection model based on two-branch deep clue perception and adaptive collaborative optimization,which can well mine the non-shared semantic clues for efficient detections.Specifically,the model first extracts fine-grained features from image salient regions and text semantic words to capture the semantic news content.Then,the heterogeneous news features are semantically aligned using a crossmodal weighted residual network,featuring on learning the shared semantic clues.For deep inference of non-shared semantic clues,the model designs an adaptive two-branch clue perception strategy to learn the cross-modal semantic correlations within the multimodal news.Specifically,the model automatically constructs an image-text clue correlation matrix based on all image regions and text words.Accordingly,a two-branch clue perception module is explicitly designed to model the probability distributions for shared and non-shared semantic correlations of all content in the matrix.On the one hand,the clue correlation optimization branch embeds an optimization algorithm to continuously update the semantic correlation boundaries of the different semantic correlation distributions iteratively,whereby the non-shared semantic clues

关 键 词：多模态虚假新闻检测 局部语义差异 跨模态语义关联 非共享语义线索 自适应协同优化 

分 类 号：TP18[自动化与计算机技术—控制理论与控制工程]