融合两级相似度的跨媒体图像文本检索  

Cross-Media Image-Text Retrieval with Two Level Similarity

作　　者：李志欣[1] 凌锋 张灿龙[1] 马慧芳 LI Zhi-xin;LING Feng;ZHANG Can-long;MA Hui-fang(Guangxi Key Laboratory of Multi-Source Information Mining and Security,Guangxi Normal University,Guilin,Guangxi 541004,China;College of Computer Science and Engineering,Northwest Normal University,Lanzhou,Gansu 730070,China)

机构地区：[1]广西师范大学广西多源信息挖掘与安全重点实验室,广西桂林541004 [2]西北师范大学计算机科学与工程学院,甘肃兰州730070

出　　处：《电子学报》2021年第2期268-274,共7页Acta Electronica Sinica

基　　金：国家自然科学基金(No.61663004,No.61966004,No.61866004,No.61762078);广西自然科学基金(No.2019GXNSFDA245018,No.2018GXNSFDA281009,No.2017GXNSFAA198365)。

摘　　要：为了更好地揭示图像和文本之间潜在的语义关联,提出了一种融合两级相似度的跨媒体检索方法,构建两个子网分别处理全局特征和局部特征,以获取图像和文本之间更好的语义匹配.图像分为整幅图像和一些图像区域两种表示,文本也分为整个语句和一些单词两种表示.设计一个两级对齐方法分别匹配图像和文本的全局和局部表示,并融合两种相似度学习跨媒体的完整表示.在MSCOCO和Flickr30K数据集上的实验结果表明,本文方法能够使图像和文本的语义匹配更准确,优于许多当前先进的跨媒体检索方法.To better reveal the latent semantic correlation between image and text,this paper proposes a cross media retrieval method by fusing two level similarity,which constructs two subnets to deal with global features and local features respectively so as to obtain better semantic matching between image and text.The image representation is divided into whole image and some image regions,and the text representation is also divided into whole sentence and some words.A two level alignment method is designed to match the global and local representation of image and text,and the two similarities are fused to learn the complete cross-media representation.The experimental results on MSCOCO and Flickr30K datasets show that the proposed method can make the semantic matching of image and text more accurate,and is superior to many state-of-the-art cross-media retrieval methods.

关 键 词：卷积神经网络 自注意力网络 两级相似度 跨媒体检索 

分 类 号：TP391[自动化与计算机技术—计算机应用技术]