Robust cross-modal retrieval with alignment refurbishment  

作　　者：Jinyi GUO Jieyu DING 

机构地区：[1]School of Computer Science and Engineering,Nanjing University of Science and Technology,Nanjing 210094,China [2]School of Mathematics and Statistics,Qingdao University,Qingdao 266071,China

出　　处：《Frontiers of Information Technology & Electronic Engineering》2023年第10期1403-1415,共13页信息与电子工程前沿(英文版)

基　　金：supported by the National Natural Science Foundation of China(No.12172186)。

摘　　要：Cross-modal retrieval tries to achieve mutual retrieval between modalities by establishing consistent alignment for different modal data.Currently,many cross-modal retrieval methods have been proposed and have achieved excellent results;however,these are trained with clean cross-modal pairs,which are semantically matched but costly,compared with easily available data with noise alignment(i.e.,paired but mismatched in semantics).When training these methods with noise-aligned data,the performance degrades dramatically.Therefore,we propose a robust cross-modal retrieval with alignment refurbishment(RCAR),which significantly reduces the impact of noise on the model.Specifically,RCAR first conducts multi-task learning to slow down the overfitting to the noise to make data separable.Then,RCAR uses a two-component beta-mixture model to divide them into clean and noise alignments and refurbishes the label according to the posterior probability of the noise-alignment component.In addition,we define partial and complete noises in the noise-alignment paradigm.Experimental results show that,compared with the popular cross-modal retrieval methods,RCAR achieves more robust performance with both types of noise.

关 键 词：Cross-modal retrieval Robust learning Alignment correction Beta-mixture model 

分 类 号：TP391[自动化与计算机技术—计算机应用技术]