A fine-grained deconfounding study for knowledge-based visual dialog  

作　　者：An-An Liu Quanhan Wu Chenxi Huang Chao Xue Xianzhu Liu Ning Xu 

机构地区：[1]School of Electrical and Information Engineering,Tianjin University,Tianjin,China [2]Tiandy Technologies Co.,Ltd,Tianjin,China [3]National and Local Joint Engineering Research Center of Space Optoelectronics Technology,Changchun University of Science and Technology,Changchun,China [4]College of Opto-Electronic Engineering,Changchun University of Science and Technology,Changchun,China

出　　处：《Visual Informatics》2024年第4期36-47,共12页可视信息学(英文)

基　　金：supported in part by the National Natural Science Foundation of China (U21B2024,62472303,62402334).

摘　　要：Knowledge-based Visual Dialog is a challenging vision-language task,where an agent engages in dialog to answer questions with humans based on the input image and corresponding commonsense knowledge.The debiasing methods based on causal graphs have gradually sparked much attention in the field of Visual Dialog(VD),yielding impressive achievements.However,existing studies focus on the coarse-grained deconfounding,which lacks a principled analysis of the bias.In this paper,we propose a fined-grained study of deconfounding on:(1)We define the confounder from two perspectives.The first is user preference(denoted as Uh),derived from human-annotated dialog history,which may introduce spurious correlations between questions and answers.The second is commonsense language bias(denoted as Uc),where certain words appear so frequently in the retrieved commonsense knowledge that the model tends to memorize these patterns,thereby establishing spurious correlations between the commonsense knowledge and the answers.(2)Given that the current question directly influences answer generation,we further decompose the confounders into Uh1,Uh2 and Uc1,Uc2,based on their relevance to the current question.Specifically,Uh1 and Uc1 represent dialog history and high-frequency words that are highly correlated with the current question,while Uh2 and Uc2 are sampled from dialog history and words with low relevance to the current question.Through a comprehensive evaluation and comparison of all components,we demonstrate the necessity of jointly considering both Uh and Uc.Fine-grained deconfounding,particularly with respect to the current question,proves to be more effective.Ablation studies,quantitative results,and visualizations further confirm the effectiveness of the proposed method.

关 键 词：Visual dialog KNOWLEDGE Deconfounding CAUSALITY 

分 类 号：G35[文化科学—情报学]