融合动量蒸馏技术的多模态情感分析的研究  

Research on a Multimodal Sentiment Analysis Model Combined with the Momentum Distillation Method

作　　者：李萌霆 武俊丽[1] LI Mengting;WU Junli(School of Information and Electronic Technology,Jiamusi University,Jiamusi Heilongjiang 154007,China)

机构地区：[1]佳木斯大学信息电子技术学院,黑龙江佳木斯154007

出　　处：《佳木斯大学学报（自然科学版）》2024年第11期18-21,91,共5页Journal of Jiamusi University：Natural Science Edition

基　　金：佳木斯大学博士专项(22ZB201515)。

摘　　要：研究多模态情感分析问题,提出一种动量蒸馏技术和多模态协同注意力机制的情感分析模型,用来解决单模态信息未充分挖掘和各模态表示存在异质性的问题。模型的多模态数据源是文本、音频和视觉数据。在各模态的单模态特征编码模块中,结合注意力机制用来突出重点信息并设计了动态调整权重模块来动态调整序列数据每个时刻不同特征维度的权重,同时在三模态环境下引入动量蒸馏技术,加入情感极性这一监督信号来促进知识蒸馏。在多模态交互模块中,引入多头注意力机制对三种模态特征进行融合,生成以文本为导向的融合表征用于最终情感分析预测。最后,在CMU-MOSI和CMU-MOSEI公开数据集上进行对比实验。The paper investigates the issue of multimodal sentiment analysis and proposes a sentiment analysis model that incorporates momentum distillation techniques and a multimodal co-attention mechanism.This model aims to address the insufficient extraction of information from single modalities and the heterogeneity present in the representations of different modalities.The multimodal data sources for the model include text,audio,and visual data.In the unimodal feature encoding module for each modality,an attention mechanism is used to highlight key information,and a dynamic weight adjustment module is designed to dynamically adjust the weights of different feature dimensions at each time step in the sequence data.Additionally,in a trimodal environment,momentum distillation is introduced,incorporating sentiment polarity as a supervisory signal to facilitate knowledge distillation.In the multimodal interaction module,a multi-head attention mechanism is employed to fuse the features of the three modalities,generating a text-guided fused representation for the final sentiment analysis prediction.Finally,comparative experiments are conducted on the publicly available CMU-MOSI and CMU-MOSEI datasets.

关 键 词：多模态情感分析 动量蒸馏技术 注意力机制 动态调整权重 

分 类 号：TP301[自动化与计算机技术—计算机系统结构]