多层CNN特征融合及多分类器混合预测的多模态虚假信息检测  

Multi-modal false information detection via multi-layer CNN-based feature fusion and multi-classifier hybrid prediction

作　　者：梁毅 吐尔地·托合提[1,2] 艾斯卡尔·艾木都拉 LIANG Yi;Turdi Tohti;Askar Hamdulla(School of Information Science and Engineering,Xinjiang University,Urumqi 830017;Xinjiang Key Laboratory of Signal Detection and Processing,Urumqi 830017,China)

机构地区：[1]新疆大学信息科学与工程学院,新疆乌鲁木齐830017 [2]新疆信号检测与处理重点实验室,新疆乌鲁木齐830017

出　　处：《计算机工程与科学》2023年第6期1087-1096,共10页Computer Engineering & Science

基　　金：国家自然科学基金(62166042,U2003207);新疆维吾尔自治区自然科学基金(2021D01C076);国防科技基金加强计划(2021-JCJQ-JJ-0059)。

摘　　要：针对现有的多模态虚假信息检测方法很少对多模态特征在特征层面进行融合,同时忽略了多模态特征后期融合作用的问题,提出了一种基于CNN多模态特征融合及多分类器混合预测的虚假信息检测模型。首次将多层CNN应用于多模态特征融合,模型首先用BERT和Swin-transformer提取文本和图像特征;随后通过多层CNN对多模态特征在特征层面进行融合,通过简单拼接对多模态特征在句子层面进行融合;最后将2种融合特征输入到不同的分类器中得到2个概率分布,并将2个概率分布按比例进行相加得到最终预测结果。该模型与基于注意力的多模态分解双线性模型(AMFB)相比,在Weibo数据集和Twitter数据集上的准确率分别提升了6.1%和4.3%。实验结果表明,所提模型能够有效提高虚假信息检测的准确率。Aiming at the problem that the existing multi-modal false information detection methods rarely fuse multi-modal features at the feature level and ignore the late fusion effect of multi-modal features,a false information detection method based on CNN multi-modal feature fusion and multi-classifier hybrid prediction is proposed.This method applies multi-layer CNN to multi-modal feature fusion for the first time.The model first uses BERT and Swin-transformer to extract text and image features,and then uses multi-layer CNN to fuse multi-modal features at the feature level.Modal features are fused at the sentence level.Finally,the two fusion features are input into different classifiers to obtain two probability distributions,and the two probability distributions are added proportionally to obtain the final prediction result.Compared with the attention-based multi-modal factorization bilinear model(AMFB),the accuracy of this model is improved by 6.1%and 4.3%on the Weibo dataset and Twitter dataset,respectively.The experimental results show that the proposed model can effectively improve the accuracy of false information detection.

关 键 词：虚假信息检测 多模态 后期融合 多层CNN 多分类器 

分 类 号：TP391.1[自动化与计算机技术—计算机应用技术]