基于两阶段缺失模态恢复的多模态情感分析方法  

Multimodal Sentiment Analysis Approach Based on Two-Stage Missing Modality Recovery

作　　者：王嫄 邓振宇 王佳鑫 张帅[1] 赵婷婷[1] 于琦 WANG Yuan;DENG Zhenyu;WANG Jiaxin;ZHANG Shuai;ZHAO Tingting;YU Qi(College of Artificial Intelligence,Tianjin University of Science and Technology,Tianjin 300457,China;College of Management,Shanxi Medical University,Taiyuan 030607,China)

机构地区：[1]天津科技大学人工智能学院,天津300457 [2]山西医科大学管理学院,太原030607

出　　处：《天津科技大学学报》2025年第1期57-63,80,共8页Journal of Tianjin University of Science & Technology

基　　金：国家自然科学基金项目(61976156)。

摘　　要：在多模态情感分析任务中,在模态缺失时存在整体多模态与单模态情感表达不一致、情感会因模态缺失而发生变化的情况。现有方法忽略缺失模态潜在的特异性语义内涵和情感表达,导致情感分析性能下降。为了解决这个问题,本文提出基于两阶段缺失模态恢复的多模态情感分析模型。首先恢复缺失模态特征,通过不变模态翻译模块恢复缺失模态与文本模态的同构信息,接着通过相似模态翻译模块在视觉词典中恢复缺失模态的异构信息,然后将已知模态和恢复的缺失模态进行融合,实现具有一致性的情感分析。在CMU-MOSI和IEMOCAP数据集上的实验表明,该模型能够有效检测和恢复缺失模态语义特征,缓解多模态与单模态情感表达不一致的问题,并具有显著的性能优势。In the task of multimodal sentiment analysis,there are cases where the overall multimodal and unimodal sentiment expressions are inconsistent in the case of missing modalities,and the sentiment will change due to missing modalities.Existing methods ignore the potential specific semantic connotations and sentiment expressions of the missing modality,resulting in degradation of sentiment analysis performance.To solve this problem,in this article we propose a multimodal sentiment analysis model based on two-stage missing modality recovery.In our proposed model,the missing modal features are first restored,by recovering the isomorphic information of the missing modality with the textual modality through the invariant modal translation module,followed by recovering the heterogeneous information of the missing modality in the visual dictionary through the similar modal translation module,and then the known modalities and the recovered missing modalities are fused to achieve consistent sentiment analysis.Experiments on CMU-MOSI and IEMOCAP showed that the model could effectively detect and recover missing modality semantic features,alleviate the problem of inconsistency between multimodal and unimodal sentiment expression,and provide significant performance advantages.

关 键 词：多模态 情感分析 模态缺失 

分 类 号：TP391.4[自动化与计算机技术—计算机应用技术]