基于注意力的多层次混合融合的多任务多模态情感分析  

Multimodal sentiment analysis based on hybrid feature fusion of multi-level attention mechanism and multi-task learning

作　　者：宋云峰 任鸽[1] 杨勇[1] 樊小超[1] Song Yunfeng;Ren Ge;Yang Yong;Fan Xiaochao(School of Computer Science&Technology,Xinjiang Normal University,Urumqi 830054,China)

机构地区：[1]新疆师范大学计算机科学技术学院,乌鲁木齐830054

出　　处：《计算机应用研究》2022年第3期716-720,共5页Application Research of Computers

基　　金：新疆维吾尔自治区自然科学基金资助项目(2021D01B72);国家自然科学基金资助项目(62066044)。

摘　　要：针对多模态情感分析中的模态内部特征表示和模态间的特征融合问题,结合注意力机制和多任务学习,提出了一种基于注意力的多层次混合融合的多任务多模态情感分析模型MAM(multi-level attention and multi-task)。首先,利用卷积神经网络和双向门控循环单元来实现单模态内部特征的提取;其次,利用跨模态注意力机制实现模态间的两两特征融合;再次,在不同层次使用自注意力机制实现模态贡献度选择;最后,结合多任务学习获得情感和情绪的分类结果。在公开的CMU-MOSEI数据集上的实验结果表明,情感和情绪分类的准确率和F;值均有所提升。Aiming at the problem of intra-modality feature representation and inter modality feature fusion in multimodal sentiment analysis, this paper proposed a multi-level hybrid fusion multi-modal sentiment analysis model based on attention mechanism and multi-task learning.Firstly, the model used convolution neural network and bi-directional gated unit to extract the single-modality internal feature.Secondly, it used the cross-modality attention mechanism to realize the pairwise feature fusion between modalities.Thirdly, it used the self-attention mechanism to select the modality contribution at different levels.Finally, combining with multi-task learning, the model obtained both sentiment and emotion classification results.The experimental results on CMU-MOSEI dataset show that this method can improve the accuracy and F;-score of sentiment and emotion classification.

关 键 词：多模态 情感分析 注意力机制 多任务学习 

分 类 号：TP391.1[自动化与计算机技术—计算机应用技术]