基于上下文感知的Transformer多模态情感分析研究  

Research on Transformer Multimodal Sentiment Analysis Based on Context Aware

作　　者：孙庆英[1] 周含 陈欣怡 刘思妍 SUN Qingying;ZHOU Han;CHEN Xinyi;LIU Siyan(School of Computer Science and Technology,Huaiyin Normal University,Huai'an 223300,China)

机构地区：[1]淮阴师范学院计算机科学与技术学院,江苏淮安223300

出　　处：《淮阴师范学院学报(自然科学版)》2025年第1期1-7,共7页Journal of Huaiyin Teachers College(Natural Science Edition)

基　　金：江苏高校“青蓝工程”优秀青年骨干教师资助项目(苏教师函[2022]29号);江苏省大数据智能工程实验室开放课题(SDGC2158);江苏省大学生创新训练计划项目(202310323103Y)。

摘　　要：旨在探索一种结合视觉和听觉信息的多模态情感识别方法,以提高情感状态检测的准确性和鲁棒性.提出了一个基于Transformer编码器和长短期记忆(Long Short-Term Memory,LSTM)网络的混合模型,用于处理和分析来自视频和音频的情感数据.首先使用Transformer编码器对多模态数据进行统一编码表示,然后利用LSTM网络对时序信息进行建模,以捕捉不同模态信息之间的交互.通过在MOSI数据集上的实验验证,提出的模型在情感识别任务上尤其是在处理复杂情感表达和非言语线索方面表现出了优越的性能.This study aims to explore a multimodal sentiment recognitionmethod that combines visual and auditory information to improve the accuracy and robustness of sentiment detection.A hybrid model based on Transformer encoder and Long Short-term Memory(LSTM)network is proposed in this paper to process and analyze emotional data from video and audio.The Transformer encoder is first used to represent the audiovisual data,and then the LSTM network is employed to model the temporal information to capture interactions between different modalities.Experimental results on the MOSI data set demonstrate the superior performance of the proposed model in sentiment recognition tasks,especially in handling complex emotional expressions and non-verbal cues.

关 键 词：多模态情感识别 深度学习 长短期记忆网络 自注意力机制 TRANSFORMER 

分 类 号：TP391.1[自动化与计算机技术—计算机应用技术]