基于一致性图卷积模型的多模态对话情绪识别  

Consistency based graph convolution network for multimodalemotion recognition in conversation

作　　者：谭晓聪 郭军军 线岩团[1,2] 相艳[1,2] Tan Xiaocong;Guo Junjun;Xian Yantuan;Xiang Yan(Faculty of Information Engineering&Automation,Kunming University of Science&Technology,Kunming 650500,China;Yunnan Key Laboratory of Artificial Intelligence,Kunming University of Science&Technology,Kunming 650500,China)

机构地区：[1]昆明理工大学信息工程与自动化学院,昆明650500 [2]昆明理工大学云南省人工智能重点实验室,昆明650500

出　　处：《计算机应用研究》2023年第10期3100-3106,共7页Application Research of Computers

基　　金：国家自然科学基金地区项目(62162037);云南省科技厅面上项目(202001AT070047,202001AT070046)。

摘　　要：多模态对话情绪识别是一项根据对话中话语的文本、语音、图像模态预测其情绪类别的任务。针对现有研究主要关注话语上下文的多模态特征提取和融合,而没有充分考虑每个说话人情绪特征利用的问题,提出一种基于一致性图卷积网络的多模态对话情绪识别模型。该模型首先构建了多模态特征学习和融合的图卷积网络,获得每条话语的上下文特征;在此基础上,以说话人在完整对话中的平均特征为一致性约束,使模型学习到更合理的话语特征,从而提高预测情绪类别的性能。在两个基准数据集IEMOCAP和MELD上与其他基线模型进行了比较,结果表明所提模型优于其他模型。此外,还通过消融实验验证了一致性约束和模型其他组成部分的有效性。Multimodal emotion recognition in conversations(MERC)is a task to predict the emotional category of the discourse in a dialogue based on its textual,audio,and visual modality.Existing studies focus on multimodal feature extraction and fusion of discourse context without fully considering the utilization of emotional features of different speakers.Therefore,this paper proposed a model of multimodal dialogue emotion recognition based on a consistent graph convolutional network.The model first constructed a graph convolutional network of multimodal feature learning and fusion,and obtained the context features of each discourse.On this basis,the average features of the speaker in the complete dialogue as the constraint to make the model learn more reasonable discourse features,so as to improve the performance of predicting emotion class.The paper compared with other baseline models on two benchmark datasets IEMOCAP and MELD.And the results show that the proposed model is superior to the other models.In addition,the paper verifies the consistency constraints and other components of the model through ablation experiments.

关 键 词：多模态 情绪识别 一致性约束 图卷积网络 情感分析 

分 类 号：TP391[自动化与计算机技术—计算机应用技术]