面向视觉问答的跨模态交叉融合注意网络  

Cross-modal chiastopic-fusion attention network for visual question answering

作　　者：王茂 彭亚雄 陆安江 WANG Mao;PENG Yaxiong;LU Anjiang(College of Big Data and Information Engineering,Guizhou University,Guiyang Guizhou 550025,China)

机构地区：[1]贵州大学大数据与信息工程学院,贵阳550025

出　　处：《计算机应用》2022年第3期854-859,共6页journal of Computer Applications

基　　金：贵州省科技成果转化项目([2017]4856)。

摘　　要：为了提高视觉问答(VQA)模型回答复杂图像问题的准确率,提出了面向视觉问答的跨模态交叉融合注意网络(CCAN)。首先,提出了一种改进的残差通道自注意方法对图像进行注意,根据图像整体信息来寻找重要区域,从而引入一种新的联合注意机制,将单词注意和图像区域注意结合在一起;其次,提出一种“跨模态交叉融合”网络生成多个特征,将两个动态信息流整合到一起,每个模态内产生有效的注意流,其中对联合特征使用逐元素相乘的方法。此外,为了避免计算成本增加,网络之间共享参数。在VQA v1.0数据集上的实验结果表明,该模型的准确率达到67.57%,较MLAN模型提高了2.97个百分点,较CAQT模型提高了1.20个百分点。所提方法有效提高了视觉问答模型的准确率,具有有效性和鲁棒性。In order to improve the accuracy of Visual Question Answering(VQA)model in answering complex image questions,a Cross-modal Chiastopic-fusion Attention Network(CCAN)for VQA was proposed.Firstly,an improved residual channel self-attention method was proposed to pay attention to the image,and to find important areas according to overall information of the image,thereby introduced a new joint attention mechanism that combined word attention and image area attention;secondly,a“cross-modal chiastopic-fusion”network was proposed to generate multiple features to integrate the two dynamic information flows together,and an effective attention flow was generated in each modal.Among them,element-wise multiplication method was used for joint features.In addition,in order to avoid an increase in computational cost,parameters were shared between networks.Experimental results on VQA v1.0 dataset show that the accuracy of the proposed model reaches 67.57%,which is 2.97 percentage points higher than that of MLAN(Multi-level Attention Network)model,1.20 percentage points higher than that of CAQT(Co-Attention network with Question Type) model.The proposed method effectively improves the accuracy of visual question answering model.The effectiveness and robustness of the method are verified.

关 键 词：视觉问答 联合注意 交叉融合 残差通道 联合特征 

分 类 号：TP391.41[自动化与计算机技术—计算机应用技术]