基于深度情感唤醒网络的多模态情感分析与情绪识别  

Deep emotional arousal network for multimodal sentiment analysis and emotion recognition

作　　者：张峰[1,2] 李希城 董春茹 花强[1,2] ZHANG Feng;LI Xi-cheng;DONG Chun-ru;HUA Qiangy(College of Mathematics and Information Sciences,Hebei University,Baoding 071002,China;Hebei Key Laboratory of Machine Learning and Computational Intelligence,Baoding 071002,China)

机构地区：[1]河北大学数学与信息科学学院,河北保定071002 [2]河北省机器学习与计算智能重点实验室,河北保定071002

出　　处：《控制与决策》2022年第11期2984-2992,共9页Control and Decision

基　　金：国家自然科学基金面上项目(61773150,61672205);河北省自然科学基金面上项目(F2018201115);河北省教育厅科学技术研究重点项目(ZD2019021);河北大学高层次创新人才科研启动经费项目。

摘　　要：随着网络平台上各类图像、视频数据的快速增长,多模态情感分析与情绪识别已成为一个日益热门的研究领域.相比于单模态情感分析,多模态情感分析中的模态融合是一个亟待解决的关键问题.受到认知科学中情感唤起模型的启发,提出一种能够模拟人类处理多通道输入信息机制的深度情感唤醒网络(DEAN),该网络可实现多模态信息的有机融合,既能处理情绪的连贯性,又能避免融合机制的选择不当而带来的问题.DEAN网络主要由以下3部分组成:跨模态Transformer模块,用以模拟人类知觉分析系统的功能;多模态BiLSTM系统,用以模拟认知比较器;多模态门控模块,用以模拟情感唤起模型中的激活结构.在多模态情感分析与情绪识别的3个经典数据集上进行的比较实验结果表明,DEAN模型在各数据集上的性能均超越了目前最先进的情感分析模型.Multimodal sentiment analysis and emotion recognition have become an increasingly popular research topic,with a large number of emerging of various images and videos on internet.Compared with the unimode sentiment analysis,one of the key issues in multimodal sentiment analysis is the fusion strategy.Inspired by the emotional arousal model in cognitive science,a deep emotional arousal network(DEAN)is proposed,which can simulate emotional coherence.The proposed DEAN consists of three parts:A crossmodal transformer module that simulates the functions of a human perception analysis system,a multimodal BiLSTM system that simulates the cognitive comparator,and a multimodal gating module that is used to simulate the activation mechanism in the physiological emotional arousal model.Extensive experimental comparisons on three benchmarks for multimodal sentiment analysis and emotion recognition are conducted,and the experimental results show that the DEAN outperforms the state-of-the-art methods.

关 键 词：多模态情感分析 多模态情绪识别 深度情感唤醒网络 跨模态Transformer 多模态BiLSTM系统 多模态门控机制 

分 类 号：TP181[自动化与计算机技术—控制理论与控制工程]