Visual commonsense reasoning with directional visual connections  

作　　者：Yahong HAN Arning WU Linchao ZHU Yi YANG 

机构地区：[1]College of Intelligence and Computing,Tianjin University,Tianjin 300350,China [2]Tianjin Key Lab of Machine Learning,Tianjin University,Tianjin 300350,China [3]School of Computer Science,University of Technology Sydney,Sydney 2007,Australia

出　　处：《Frontiers of Information Technology & Electronic Engineering》2021年第5期625-637,共13页信息与电子工程前沿（英文版）

基　　金：Project supported by the National Natural Science Foundation of China(Nos.61876130 and 61932009)。

摘　　要：To boost research into cognition-level visual understanding,i.e.,making an accurate inference based on a thorough understanding of visual details,visual commonsense reasoning(VCR)has been proposed.Compared with traditional visual question answering which requires models to select correct answers,VCR requires models to select not only the correct answers,but also the correct rationales.Recent research into human cognition has indicated that brain function or cognition can be considered as a global and dynamic integration of local neuron connectivity,which is helpful in solving specific cognition tasks.Inspired by this idea,we propose a directional connective network to achieve VCR by dynamically reorganizing the visual neuron connectivity that is contextualized using the meaning of questions and answers and leveraging the directional information to enhance the reasoning ability.Specifically,we first develop a GraphVLAD module to capture visual neuron connectivity to fully model visual content correlations.Then,a contextualization process is proposed to fuse sentence representations with visual neuron representations.Finally,based on the output of contextualized connectivity,we propose directional connectivity to infer answers and rationales,which includes a ReasonVLAD module.Experimental results on the VCR dataset and visualization analysis demonstrate the effectiveness of our method.

关 键 词：Visual commonsense reasoning Directional connective network Visual neuron connectivity Contextualized connectivity Directional connectivity 

分 类 号：TP181[自动化与计算机技术—控制理论与控制工程]