基于图像相对位置和负向感知的图文匹配  

Image-text matching based on image relative position and negative perception

作　　者：余超 王铭硕 赵子樵 于清[2] YU Chao;WANG Mingshuo;ZHAO Ziqiao;YU Qing(School of Software,Xinjiang University,Urumqi 830046,China;School of Information Science and Engineering,Xinjiang University,Urumqi 830046,China)

机构地区：[1]新疆大学软件学院,新疆乌鲁木齐830046 [2]新疆大学信息科学与工程学院,新疆乌鲁木齐830046

出　　处：《现代电子技术》2024年第17期88-93,共6页Modern Electronics Technique

基　　金：国家973重点研发计划项目(2014CB340506)。

摘　　要：图文匹配任务在计算机视觉以及多模态信息处理领域引起了广泛关注。这一跨模态任务主要难点在于如何高效地提取视觉和文本的信息以及如何解决不一致图文冲突问题。文中提出了一种新颖的图像文本匹配方法,利用图像对象相对位置的注意力机制解决忽视图像中物体相对位置信息的问题,从而更好地关注视觉信息的提取,同时为了解决忽视图像-文本间不对齐内容相似度贡献的问题,运用了负向感知模块关注物体相对位置和文本中的单词非对齐的信息对相似度的负贡献,从而提高图像-文本匹配相似度的准确性。最后,在公开图文匹配数据集Flickr30K上达到了最好的r_(Sum),比当前最好的负向感知模型提高了7.3,取得了最先进的性能。Image-text matching has attracted extensive attention in the field of computer vision and multimodal information processing.The challenges of the cross-modal task are how to extract the information of vision and text efficiently and how to deal with the inconsistency image-text conflicts.In this paper,a novel image-text matching method is proposed.In this method,the attention mechanism of the relative position of image objects is utilized to avoid ignoring the relative position information of objects in an image,so as to extract visual information more efficiently.Meanwhile,in order to get rid of ignoring the similarity contribution of the unaligned content between image and text,a negative perception module is employed to pay attention to the negative contribution from the relative position of the image objects and the information of unaligned words in the text,so as to improve the accuracy of image-text matching.Finally,the proposed method achieves the best r_(Sum) score on the publicly available image-text matching dataset Flickr30K,and it is an improvement of 7.3%over the best negative perception model at present.Therefore,the proposed method is of the most advanced performance.

关 键 词：多模态 图文匹配 视觉信息提取 相对位置编码 注意力机制 负向感知 

分 类 号：TN911.73-34[电子电信—通信与信息系统] TP391[电子电信—信息与通信工程]