视觉知识:跨媒体智能进化的新支点  

The review of visual knowledge:a new pivot for cross-media intelligence evolution

作　　者：杨易[1] 庄越挺[1] 潘云鹤[1,2] Yang Yi;Zhuang Yueting;Pan Yunhe(College of Computer Science and Technology,Zhejiang University,Hangzhou 310027,China;Zhejiang Laboratory,Hangzhou 310027,China)

机构地区：[1]浙江大学计算机科学与技术学院,杭州310027 [2]之江实验室,杭州310027

出　　处：《中国图象图形学报》2022年第9期2574-2588,共15页Journal of Image and Graphics

基　　金：国家重点研发计划资助(2020AAA0108800);中央高校基本科研业务费专项资金资助(226-2022-00051)。

摘　　要：回顾跨媒体智能的发展历程,分析跨媒体智能的新趋势与现实瓶颈,展望跨媒体智能的未来前景。跨媒体智能旨在融合多来源、多模态数据,并试图利用不同媒体数据间的关系进行高层次语义理解与逻辑推理。现有跨媒体算法主要遵循了单媒体表达到多媒体融合的范式,其中特征学习与逻辑推理两个过程相对割裂,无法综合多源多层次的语义信息以获得统一特征,阻碍了推理和学习过程的相互促进和修正。这类范式缺乏显式知识积累与多级结构理解的过程,同时限制了模型可信度与鲁棒性。在这样的背景下,本文转向一种新的智能表达方式——视觉知识。以视觉知识驱动的跨媒体智能具有多层次建模和知识推理的特点,并易于进行视觉操作与重建。本文介绍了视觉知识的3个基本要素,即视觉概念、视觉关系和视觉推理,并对每个要素展开详细讨论与分析。视觉知识有助于实现数据与知识驱动的统一框架,学习可归因可溯源的结构化表达,推动跨媒体知识关联与智能推理。视觉知识具有强大的知识抽象表达能力和多重知识互补能力,为跨媒体智能进化提供了新的有力支点。We review the recent development of cross-media intelligence,analyze its new trends and challenges,and discuss future prospects of cross-media intelligence.Cross-media intelligence is focused on the integration of multi-source and multi-modal data.It attempts to use the relationship between different media data for high-level semantic understanding and logical reasoning.Existing cross-media algorithms mainly follow the paradigm of“single media representation”to“multimedia integration”,in which the two processes of feature learning and logical reasoning are relatively disconnected.It is unlikely to synthesize multi-source and multi-level semantic information to obtain unified features,which hinders the mutual benefits of the reasoning and learning process.This paradigm is lack of the process of explicit knowledge accumulation and multi-level structure understanding.At the same time,it restricts the interpretability and robustness of the model.We interpret new representation method,i.e.,visual knowledge.Visual knowledge driven cross-media intelligence has the features of multi-level modeling and knowledge reasoning.Its built-in mechanisms can implement operations and reconstruction visually,which learns knowledge alignment and association.To establish a unified way of knowledge representation learning,the theory of visual knowledge has been illustrated as mentioned below:1)we introduce three key factors of visual contexts,i.e.,concept,visual relationship,and visual reasoning.Visual knowledge has capable of knowledge representations abstraction and multiple knowledge complementing.Visual relations represent the relationship between visual concepts and provide an effective basis for more complex cross-media visual reasoning.We demonstrate visual-based spatio-temporal and causal relationships,but the visual relationship is not limited to these categories.We recommend that the pairwise visual relationships should be extended to multi-objects cascade relationships and the integrated spatio-temporal and causal re

关 键 词：跨媒体智能 视觉知识 视觉概念 视觉关系 视觉推理 

分 类 号：TP391.7[自动化与计算机技术—计算机应用技术]