采用目标注意力的方面级多模态情感分析研究  

Aspect-level multimodal sentiment analysis via object-attention

作　　者：朱超杰 闫昱名 初宝昌 李刚 黄河燕[1] 高小燕 ZHU Chaojie;YAN Yuming;CHU Baochang;LI Gang;HUANG Heyan;GAO Xiaoyan(School of Computer Science&Technology,Beijing Institute of Technology,Beijing 100081,China;Beijing Huadian E-Commerce Technology Co.,Ltd.,Beijing 100073,China;Faculty of Information Technology,Beijing University of Technology,Beijing 100124,China)

机构地区：[1]北京理工大学计算机学院,北京100081 [2]北京华电电子商务科技有限公司,北京100073 [3]北京工业大学计算机学院,北京100124

出　　处：《智能系统学报》2024年第6期1562-1572,共11页CAAI Transactions on Intelligent Systems

基　　金：国家自然科学基金项目(U21B2009);横向科技项目(2023110051000823).

摘　　要：方面级的多模态情感分析(aspect-level multimodal sentiment analysis,ALMSA)旨在识别出语句和图像信息在某个特定方面上所表现出的情感极性。该任务现有分析模型使用的均是图像的全局特征,并未考虑原始图像信息中的细节信息。针对这一问题,提出一种基于目标注意力的方面级多模态情感分析模型OABALMSA(object-attention based aspect-level multimodal sentiment analysis)。采用目标检测算法捕获原始图像中目标的细节信息;引入目标注意力机制并构建迭代的融合层来完成多模态信息的充分融合;针对数据较高的复杂性所导致的训练困难问题,为模型制定课程式学习策略。经课程式学习训练的OAB-ALMSA模型在TWITTER-2015数据集上得到了最高的F1,这表明对图像中细节信息的利用能够提高模型对数据的综合理解,提升预测效果。Aspect-level multimodal sentiment analysis(ALMSA)aims to identify the sentiment polarity of a specific aspect word using both sentence and image data.Current models often rely on the global features of images,overlooking the details in the original image.To address this issue,we propose an object attention-based aspect-level multimodal sentiment analysis model(OAB-ALMSA).This model first employs an object detection algorithm to capture the detailed information of the objects from the original image.It then applies an object-attention mechanism and builds an iterative fusion layer to fully fuse the multimodal information.Finally,a curriculum learning strategy is developed to tackle the challenges of training with complex samples.Experiments conducted on TWITTER-2015 data sets demonstrate that OAB-ALMSA,when combined with curriculum learning,achieves the highest F1.These results highlight that leveraging detailed image data enhances the model’s overall understanding and improves prediction accuracy.

关 键 词：方面级情感分析 多模态 情感分析 目标检测 自注意力机制 自然语言处理 深度学习 特征提取 

分 类 号：TP391[自动化与计算机技术—计算机应用技术]