基于分层融合策略和上下文信息嵌入的多模态情绪识别  

Multimodal Emotion Recognition Based on Hierarchical Fusion Strategy and Contextual Information Embedding

作　　者：孙明龙 欧阳纯萍[1] 刘永彬[1] 任林[1] SUN Minglong;OUYANG Chunping;LIU Yongbin;REN Lin(School of Computing,University of South China,Hengyang 421200)

机构地区：[1]南华大学计算机学院,衡阳421200

出　　处：《北京大学学报（自然科学版）》2024年第3期393-402,共10页Acta Scientiarum Naturalium Universitatis Pekinensis

基　　金：湖南省自然科学基金(2022JJ30495);湖南省教育厅重点科研项目(22A0316)资助。

摘　　要：现有的多模态融合策略大多将不同模态特征进行简单拼接,忽略了针对单个模态固有特点的个性化融合需求。同时,在情绪识别阶段,独立地看待单个话语的情绪而不考虑其在前后话语语境下的情绪状态,可能导致情绪识别错误。为了解决上述问题,提出一种基于分层融合策略和上下文信息嵌入的多模态情绪识别方法,通过分层融合策略,采用层次递进的方式,依次融合不同的模态特征,以便减少单个模态的噪声干扰并解决不同模态间表达不一致的问题。该方法还充分利用融合后模态的上下文信息,综合考虑单个话语在上下文语境中的情绪表示,以便提升情绪识别的效果。在二分类情绪识别任务中,该方法的准确率比SOTA模型提升1.54%。在多分类情绪识别任务中,该方法的F1值比SOTA模型提升2.79%。Existing fusion strategies often involve simple concatenation of modal features,disregarding persona-lized fusion requirements based on the characteristics of each modality.Additionally,solely considering the emo-tions of individual utterances in isolation,without accounting for their emotional states within the context,can lead to errors in emotion recognition.To address the aforementioned issues,this paper proposes a multimodal emotion recognition method based on a layered fusion strategy and the incorporation of contextual information.The method employs a layered fusion strategy,progressively integrating different modal features in a hierarchical manner to re-duce noise interference from individual modalities and address inconsistencies in expression across different mo-dalities.It leverages the contextual information to comprehensively analyze the emotional representation of each utterance within the context,enhancing overall emotion recognition performance.In binary emotion classification tasks,the proposed method achieves a 1.54%improvement in accuracy compared with the state-of-the-art(SOTA)model.In multi-class emotion recognition tasks,the F1 score is improved by 2.79%compared to SOTA model.

关 键 词：分层融合 噪声干扰 上下文信息嵌入 

分 类 号：TP391.41[自动化与计算机技术—计算机应用技术] TP391.1[自动化与计算机技术—计算机科学与技术] TN912.3[电子电信—通信与信息系统]