基于布局化-语义联合表征遥感图文检索方法  

Remote sensing image-text retrieval based on layout semantic joint representation

作　　者：张若愚 聂婕[1] 宋宁 郑程予 魏志强[1] ZHANG Ruoyu;NIE Jie;SONG Ning;ZHENG Chengyu;WEI Zhiqiang(Department of Information Science and Engineering,Ocean University of China,Qingdao 266101,China)

机构地区：[1]中国海洋大学信息科学与工程学部,青岛266101

出　　处：《北京航空航天大学学报》2024年第2期671-683,共13页Journal of Beijing University of Aeronautics and Astronautics

基　　金：国家重点研发计划(2021YFF070400);国家自然科学基金(62072418,62172376);中央高校基本科研业务费专项资金(202042008)。

摘　　要：遥感图文检索可以从类别繁多、内容复杂的遥感数据中检索到有价值的信息,对环境评估、城市规划以及灾害预测具有重要意义。但是,遥感图文跨模态检索存在一个关键问题,即遥感图像的空间布局信息被忽略。其主要体现在2个方面:①遥感目标的远距离建模困难;②遥感相邻次要目标被淹没。基于以上问题,提出了一种基于布局化-语义联合表征的跨模态遥感图像文本检索(SL-SJR),主要包括主导语义监督的布局化视觉特征提取(DSSL)模块、布局化视觉-全局语义交叉指导(LV-GSCG)模块和多视角匹配(MVM)模块。DSSL模块实现主导语义类别特征监督下图像的布局化建模。LV-GSCG模块计算布局化视觉特征与文本中提取的全局语义特征的相似度来实现不同模态特征的交互。MVM模块建立跨模态特征指导的多视角度量匹配机制以消除跨模态数据之间的语义鸿沟。在4个基线遥感图像文本数据集上的实验验证,结果表明所提方法在大多数跨模态遥感图像文本检索任务中可以达到最先进的性能。Remote sensing image-text retrieval can retrieve valuable information from remote sensing data.It is of great significance to environmental assessment,urban planning and disaster prediction.However,there is a key problem that the spatial layout information of remote sensing images is ignored,which is mainly reflected in two aspects:one is the difficulty of long-distance modeling of remote sensing targets;the other,the submerge of the remote sensing adjacent secondary targets.Based on the above problems,this paper proposes a cross modal remote sensing image-text retrieval model based on layout semantic joint representation,which includes the dominant semantic supervison layout visual feature extraction module(DSSL),Layout visual-global semantic cross guidance(LV-GSCG)and multi-view matching(MVM).The DSSL module realizes the layout modeling of images under the supervision of dominant semantic category features.The LV-GSCG module calculates the similarity between the layout visual features and the global semantic features extracted from text to realize the interaction of different modal features.The MVM module establishes a cross-modal feature-guided multi-view metric matching mechanism to eliminate the semantic gap between the cross-modal data.Experimental validation on four baseline remote sensing image text datasets shows that the model can achieve state-of-the-art performance in most cross-modal remote sensing image text retrieval tasks.

关 键 词：遥感图像 跨模态检索 空间布局信息 主导语义监督 类监督机制 

分 类 号：P407.8[天文地球—大气科学及气象学]