面向科技资讯的基于语义对抗和媒体对抗的跨媒体检索方法  

Scientific and Technological Information Oriented Semantics-Adversarial and Media-Adversarial Based Cross-Media Retrieval Method

作　　者：李昂 杜军平[1,2] 寇菲菲[1,2] 薛哲 徐欣 许明英 姜阳 Li Ang;Du Junping;Kou Feifei;Xue Zhe;Xu Xin;Xu Mingying;Jiang Yang(School of Computer Science(National Pilot School of Software Engineering),Beijing University of Posts and Telecommunications,Beijing 100876;Beijing Key Laboratory of Intelligent Telecommunication Software and Multimedia(Beijing University of Posts and Telecommunications),Beijing 100876)

机构地区：[1]北京邮电大学计算机学院(国家示范性软件学院),北京100876 [2]智能通信软件与多媒体北京市重点实验室(北京邮电大学),北京100876

出　　处：《计算机研究与发展》2023年第11期2660-2670,共11页Journal of Computer Research and Development

基　　金：国家自然科学基金重大项目(62192784);第八届青年人才托举工程项目(2022QNRC001)。

摘　　要：科技资讯跨媒体检索是跨媒体领域的重要任务之一,面临着多媒体数据间异构鸿沟和语义鸿沟亟待打破的难题.通过跨媒体科技资讯检索,用户能够从多源异构的海量科技资源中获取目标科技资讯.这有助于设计出符合用户需求的应用,包括科技资讯推荐、个性化科技资讯检索等.跨媒体检索研究的核心是学习一个公共子空间,使得不同媒体的数据在该子空间中可以直接相互比较.在子空间学习中,现有方法往往聚焦于建模媒体内数据的判别性和媒体间数据在映射后的不变性,却忽略了媒体间数据在映射前后的语义一致性和语义内的媒体判别性,使得跨媒体检索效果存在局限性.鉴于此,提出一种面向科技资讯的基于语义对抗和媒体对抗的跨媒体检索方法(SMCR),寻找可供映射的有效公共子空间.具体而言,SMCR在建模媒体内语义判别性之外,将媒体间语义一致性损失最小化,以保留映射前后的语义相似性.此外,SMCR构建基础特征映射网络和精炼特征映射网络,联合最小化语义内的媒体判别性损失,有效增强了特征映射网络混淆媒体判别网络的能力.在2个数据集上的大量实验结果表明,所提出的SMCR方法在跨媒体检索中的表现优于最前沿的方法.Cross-media retrieval of scientific and technological information is one of the important tasks in the crossmedia study.Cross-media scientific and technological information retrieval obtains target information from massive multi-source and heterogeneous scientific and technological resources,which helps to design applications that meet users' needs,including scientific and technological information recommendation,personalized scientific and technological information retrieval,etc.The core of cross-media retrieval is to learn a common subspace,in which data from different media can be directly compared with each other.In subspace learning,existing methods often focus on modeling the discrimination of intra-media data and the invariance of inter-media data after mapping,while ignoring semantic consistency within media and media discrimination within semantics,which limits the result of cross-media retrieval.In light of this,we propose a scientific and technological information oriented semanticsadversarial and media-adversarial cross-media retrieval method(SMCR) to find an effective common subspace.Specifically,SMCR minimizes the loss of inter-media semantic consistency in addition to modeling intra-media semantic discrimination,to preserve semantic similarity before and after mapping.Furthermore,SMCR constructs a basic feature mapping network and a refined feature mapping network to jointly minimize the media discriminative loss within semantics,to enhance the feature mapping network's ability to confuse the media discriminant network.Experimental results on two datasets demonstrate that the proposed SMCR outperforms state-of-the-art methods in cross-media retrieval.

关 键 词：跨媒体检索 对抗学习 科技资讯 媒体约束 语义一致性 

分 类 号：TP391[自动化与计算机技术—计算机应用技术]