基于多层LSTM融合的多模态情绪识别  

Multi-modal Emotion Recognition Based on Multi-LSTMs Fusion

作　　者：张亚伟 吴良庆 王晶晶[1] 李寿山[1] ZHANG Yawei;WU Liangqing;WANG Jingjing;LI Shoushan(School of Computer Science and Technology,Soochow University,Suzhou,Jiangsu 215006,China)

机构地区：[1]苏州大学计算机科学与技术学院,江苏苏州215006

出　　处：《中文信息学报》2022年第5期145-152,共8页Journal of Chinese Information Processing

基　　金：国家自然科学基金(62006166,61976146,62076176);中国博士后科学基金(2019M661930);江苏高校优势学科建设工程自主项目。

摘　　要：情绪分析一直是自然语言处理领域的研究热点,而多模态情绪分析是当前该领域的一个挑战。已有研究在上下文信息和不同模态时间序列信息交互方面存在不足,该文提出了一个新颖的多层LSTM融合模型(Multi-LSTMs Fusion Model, MLFN),通过分层LSTM分别设置单模态模内特征提取层、双模态和三模态模间融合层进行文本、语音和图像三个模态之间的深度融合,在考虑模态内部信息特征的同时深度捕获模态之间的交互信息。实验结果表明,基于多层LSTM多模态融合网路能够较好地融合多模态信息,大幅度提升多模态情绪识别的准确率。Sentiment analysis is a popular research issue in the field of natural language processing, and multimodal sentiment analysis is the current challenge in this task. Existing studies are defected in capturing context information and combining information streams of different models. This paper proposes a novel multi-LSTMs Fusion Model Network(MLFN), which performs deep fusion between the three modalities of text, voice and image via the internal feature extraction layer for single-modal, and the inter-modal fusion layer for dual-modal and tri-modal. This hierarchical LSTM framework takes into account the information features inside the modal while capturing the interaction between the modals. Experimental results show that the proposed method can better integrate multi-modal information, and significantly improve the accuracy of multi-modal emotion recognition.

关 键 词：多模态 情绪分析 LSTM 

分 类 号：TP391[自动化与计算机技术—计算机应用技术]