基于跨尺度特征融合自注意力的图像描述方法  

Cross-scale Feature Fusion Self-attention for Image Captioning

作　　者：王鸣展 冀俊忠 贾奥哲[1,2] 张晓丹 WANG Ming-zhan;JI Jun-zhong;JIA Ao-zhe;ZHANG Xiao-dan(School of Computer Science,Faculty of Information Technology,Beijing University of Technology,Beijing 100124,China;Beijing Institute of Artificial Intelligence,Beijing University of Technology,Beijing 100124,China)

机构地区：[1]北京工业大学信息学部计算机学院,北京100124 [2]北京工业大学北京人工智能研究院,北京100124

出　　处：《计算机科学》2022年第10期191-197,共7页Computer Science

基　　金：国家自然科学基金(61906007);北京市教育委员会科学研究计划(KM202110005022,KZ202210005009)。

摘　　要：近年来,基于自注意力机制的编码器-解码器框架已经成为主流的图像描述模型。然而,编码器中的自注意力只建模低尺度特征的视觉关系,忽略了高尺度视觉特征中的一些有效信息,从而影响了生成描述的质量。针对该问题,文中提出了一种基于跨尺度特征融合自注意力的图像描述方法。该方法在进行自注意力运算时,将低尺度和高尺度的视觉特征进行跨尺度融合,从视觉角度上提高自注意力关注的范围,增加有效视觉信息,减少噪声,从而学习到更准确的视觉语义关系。在MS COCO数据集上的实验结果表明,所提方法能够更精确地捕获跨尺度视觉特征间的关系,生成更准确的描述。特别地,该方法是一种通用的方法,通过与其他基于自注意力的图像描述方法相结合,能进一步提高模型性能。In recent years, the encoder-decoder framework based on self-attention mechanism has become the mainstream model in image captioning.However, self-attention in the encoder only models the visual relations of low-scale features, ignoring some effective information in high-scale visual features, thus affecting the quality of the generated descriptions.To solve this problem, this paper proposes a cross-scale feature fusion self-attention(CFFSA) method for image captioning.Specifically, CFFSA integrates low-scale and high-scale visual features in self-attention to improve the range of attention from a visual perspective, which increases effective visual information and reduces noise, thereby learning more accurate visual and semantic relationships.Experiments on MS COCO dataset show that the proposed method can more accurately capture the relationship between cross-scale visual features and generate more accurate descriptions.In addition, CFFSA is a general method, which can further improve the performance of the model by combining with other self-attention based image captioning methods.

关 键 词：图像描述 自注意力 跨尺度特征融合 

分 类 号：TP181[自动化与计算机技术—控制理论与控制工程]