动态时间序列建模的多模态情感识别方法  

Multimodal Emotion Recognition Method Based on Dynamic Time Sequence Modeling

作　　者：李佳泽 梅红岩 贾丽云 李文娅 LI Jiaze;MEI Hongyan;JIA Liyun;LI Wenya(College of Electronic and Information Engineering,Liaoning University of Technology,Jinzhou,Liaoning 121001,China;College of Software,Liaoning University of Technology,Jinzhou,Liaoning 121001,China)

机构地区：[1]辽宁工业大学电子与信息工程学院,辽宁锦州121001 [2]辽宁工业大学软件学院,辽宁锦州121001

出　　处：《计算机工程与应用》2025年第1期196-205,共10页Computer Engineering and Applications

基　　金：国家自然基金面上项目(62273170);辽宁省教育厅科研项目(JZL202015404,LJKZ0625);辽宁省教育厅面上项目(JYTMS20230869)。

摘　　要：现有的情感识别研究未充分考虑语音信号中的局部-全局信息和长期时间依赖关系,文本特征提取也存在特征稀疏和信息丢失的问题。为解决上述问题,提出动态时间序列建模的多模态情感识别方法。设计动态时间窗口模块分割语音信号从而捕捉局部-全局信息,并通过双向序列建模捕获信号中的空间信息。考虑到文本信息对情感分析的重要性,采用基于Transformer模型的卷积神经网络捕捉文本中不同位置间的依赖关系建模较长的上下文信息,最后将两种模态进行融合得到最终的情感分类。模型在IEMOCAP数据集上的实验结果表明,相比其他主流模型具有更好的多模态情感识别效果。Existing emotion recognition studies have not fully considered the local-global information and long-term time dependencies in speech signals,and text feature extraction also suffers from feature sparsity and information loss.To solve the above problems,multimodal emotion recognition method based on dynamic time sequence modeling is proposed.The dynamic time window module is designed to segment the speech signal so as to capture the local-global information,and the spatial information in the signal is captured by bi-directional sequence modelling.Considering the importance of text information for emotion analysis,a convolutional neural network based on the Transformer model is used to capture the longer contextual information by modelling the dependencies between different locations in the text,and finally the two modalities are fused to obtain the final emotion classification.The experimental results of the model on the IEMOCAP dataset show better multimodal emotion recognition compared to other mainstream models.

关 键 词：多模态情感分析 动态时间窗口 双向时间序列建模 卷积神经网络 多模态融合 

分 类 号：TP391[自动化与计算机技术—计算机应用技术]