基于多视图关注网络的图文多模态情感分析模型  

A graphic text multimodal sentiment analysis model based on multiview attention network

作　　者：丛子涵 张思佳 CONG Zihan;ZHANG Sijia(Liaoning Provincial Key Laboratory of Marine Information Technology,College of Information Engineering,Dalian Ocean University,Dalian 116023,China;MOE Key Laboratory of Environment Controlled Aquaculture,Dalian Ocean University,Dalian 116023,China;Dalian Key Laboratory of Smart Fisheries,Dalian 116023,China)

机构地区：[1]大连海洋大学信息工程学院辽宁省海洋信息技术重点实验室,辽宁大连116023 [2]大连海洋大学设施渔业教育部重点试验室,辽宁大连116023 [3]大连市智慧渔业重点实验室,辽宁大连116023

出　　处：《现代电子技术》2024年第12期157-164,共8页Modern Electronics Technique

基　　金：辽宁省教育厅高等学校基本科研项目面上项目(LJKMZ20221095);辽宁省重点研发计划项目(2023JH26/10200015)。

摘　　要：针对现有多模态情感分类模型无法全面、准确地捕获复杂的情感信息,以及融合过程中没有充分挖掘两者之间的潜在关联,导致模型结构冗余复杂、计算效率低下的问题,提出一种多视图关注网络(MPF-Net)模型。该模型通过引入多维感知特征捕获机制,全面而精确地获取图像和文本中蕴含的情感信息;其次,采用增强的记忆互动学习机制,使模型能够更加有效地提取和融合单模态特征,并在多轮迭代中不断更新和优化这些特征,从而捕捉到更深层次的情感细节;再构建一个高级深度学习框架,该框架采用生成对抗网络(GAN)与池化技术的深度融合单元,以实现复杂数据特征的高效提取与整合;最后,在保留原有特征信息的基础上进行特征整合,同时通过降维技术降低模型的复杂性,提高计算效率。在公开数据集MVSA-Single和MVSA-Multiple以及自建数据集上通过实验验证所提模型的准确性,结果表明,与多个基线模型对比,所提模型的准确率和F1值均有所提高。In allusion to the problems that the existing multimodal emotion classification models cannot comprehensively and accurately capture the complex emotion information as well as the fusion process does not fully explore the potential correlation between the two,which leads to the redundant and complex model structure and computational inefficiency,a multi-view concern network(MPF-Net)model is proposed.In this model,multidimensional perceptual feature capture mechanism is introduced to comprehensively and accurately acquire the emotional information embedded in images and texts.An enhanced memory-interactive learning mechanism is employed to make the model effectively extract and fuse single modal features,and continuously update and optimize these features in multiple iterations,thereby capturing deeper emotional details.An advanced deep learning framework is constructed,which can employ a deep fusion unit of generative adversarial network(GAN)and pooling techniques for efficient extraction and integration of complex data features.The feature integration is performed on the basis of retaining the original feature information,while the complexity of the model is reduced and the computational efficiency is improved by means of the dimensionality reduction techniques.The accuracy of the proposed model was verified by experiments on public datasets MVSA-Single,MVSA-Multiple,and self-built datasets.The results show that,in comparison with multiple baseline models,the accuracy and F1 value of the proposed model is improved.

关 键 词：多模态情感分析 对抗学习 多视图网络 生成对抗网络 文本特征提取 特征融合 

分 类 号：TN911.25-34[电子电信—通信与信息系统] TP391.1[电子电信—信息与通信工程]