基于共同子空间分类学习的跨媒体检索研究  

Study on Cross-media Information Retrieval Based on Common Subspace Classification Learning

作　　者：韩红旗[1,2] 冉亚鑫 张运良 桂婕[1] 高雄 易梦琳[1,2] HAN Hong-qi;RAN Ya-xin;ZHANG Yun-liang;GUI Jie;GAO Xiong;YI Meng-lin(Institute of Scientific and Technical Information of China,Beijing 100038,China;Key Laboratory of Rich-media Knowledge Organization and Service of Digital Publishing Content,National Press and Publication Administration,Beijing 100038,China)

机构地区：[1]中国科学技术信息研究所,北京100038 [2]富媒体数字出版内容组织与知识服务重点实验室(国家新闻出版署),北京100038

出　　处：《计算机科学》2022年第5期33-42,共10页Computer Science

基　　金：中国科学技术信息研究所重点工作项目(ZD2020-09);国家自然科学基金(71473237)。

摘　　要：不同媒体数据间由于存在严重的异构鸿沟和语义鸿沟,而不能直接计算它们之间的语义相似度,从而影响了跨媒体检索的实现和效果。当前提出的共同子空间学习虽能实现跨媒体语义关联和检索,但多采用一般的特征提取技术,且在语义匹配时的分类效果较差,不能有效实现跨媒体数据的高层语义关联计算,影响了检索效果。对此,提出Stacking-DSCM-WR跨媒体关联方法,用于文档和图像之间的跨媒体检索。该方法基于词向量技术形成文档的特征表示向量,通过残差网络技术抽取图像的特征表示向量,采用深度典型相关性分析技术将不同模态的数据投影到共同子空间下,然后采用Stacking集成学习算法获取文本和图像在同一高层概念语义空间上的分布,使得两种不同模态的数据可以进行语义匹配、相似性计算。在Wikipedia和Pascal Sentence两个小型跨媒体数据集和一个较大规模跨媒体数据集INRIA-Websearch上分别开展跨媒体检索实验,证实了所提方法能够有效地抽取文本和图像的特征,实现跨媒体数据在高层语义空间上的关联和匹配,与相近跨媒体检索方法在MAP指标上的对比显示,该方法能够取得较好的检索效果。The semantic similarity between two different media data can not be calculated directly because of the serious heterogeneous gap and semantic gap between them,which affects the implementation and effect of cross media retrieval.Although the common space learning can achieve cross media semantic association and retrieval,the retrieval performance is not satisfied.The main reason is that it uses common feature extraction technology and general classification algorithm to implement semantic correlation and match.Aiming at this problem,the study proposes a novel cross media correlation method called Stacking-DSCM-WR for cross media retrieval between documents and images.WR means that text feature extraction is based on word-embedding technique and the image feature extraction is based on ResNet technique.DSCM means that the deep semantic correlation and match technology is exploited to project data of different modalities into a common subspace.Stacking is a kind of ensemble lear-ning algorithm.It is employed to produce the distribution of text documents and images on the same high-level conceptual semantic space for cross-media retrieval.The experiments are carried out on two smaller cross-media datasets,Wikipedia and Pascal Sentence,and one larger cross-media dataset,INRIA-Websearch,respectively.The results show that the proposed method can effectively extract the features of text and image,and realize the correlation and match of cross media data in high-level semantic space.The comparisons with similar cross media retrieval methods show that the proposed method achieves the best retrieval effect based on MAP metric.

关 键 词：跨媒体信息检索 语义关联 集成学习 词向量 残差网络 

分 类 号：TP391[自动化与计算机技术—计算机应用技术] G354[自动化与计算机技术—计算机科学与技术]