基于高阶模式间关系的跨模态关联学习  

Cross-modal association learning based on higher-order inter-modalities relationships

作　　者：康娟[1] 汪传建[1] KANG Juan;WANG Chuan-jian(College of Information Science and Technology,Shihezi University,Shihezi 832000,China)

机构地区：[1]石河子大学信息科学与技术学院

出　　处：《计算机工程与设计》2020年第1期265-270,共6页Computer Engineering and Design

基　　金：国家自然科学基金项目(61741210);石河子大学应用基础研究青年基金项目(ZZZC201609)

摘　　要：针对多模态数据查询和基于模式之间低阶关系的查询方法存在的局限性,提出一种基于高阶模式之间关系的跨模态关联学习模型。在超网络的基础上,构建一种具有层次结构的分层超网络模型,由超网络数目等于模式数目的模式层和代表几种模式之间关系的集成层构成;经过训练的分层超网络就可以通过跨模态关联推理生成对于给定多模态查询的文本术语和视觉词关键字,实现多模态查询。基于大量带有图像文章的实验结果表明,提出模型可以提高生成关键字的相似性,生成文本术语,成功检索出具有小部分信息的文章。In view of multi-modal data query and the limitations of the query methods based on low-order inter-modalities relationships,a cross-modal association learning model based on higher-order inter-modalities relationships was proposed.A super-network model with hierarchical structures was constructed on the basis of super-network.It consisted of a modality layer with the number of super-networks equal to the number of modalities and an integration layer representing the relationship between seve-ral modalities.A trained hierarchical super-networks generated text terms and visual word keywords for given multimodal queries by cross-modal association reasoning,to realize multimodal queries.Experimental results based on a large number of articles with images show that the proposed model can not only improve the similarity of the generated keywords,but generate the text terms and retrieve the article with small part of information.

关 键 词：多媒体数据 信息查询 分层超网络 跨模态 关联学习 相似性 

分 类 号：TP319[自动化与计算机技术—计算机软件与理论]