MKEAH:Multimodal knowledge extraction and accumulation based on hyperplane embedding for knowledge-based visual question answering  

作　　者：Heng ZHANG Zhihua WEI Guanming LIU Rui WANG Ruibin MU Chuanbao LIU Aiquan YUAN Guodong CAO Ning HU 

机构地区：[1]Tongji University,Shanghai 2000092,China [2]Alibaba Group,Shanghai 200030,China

出　　处：《虚拟现实与智能硬件（中英文）》2024年第4期280-291,共12页Virtual Reality & Intelligent Hardware

基　　金：Supported by National Nature Science Foudation of China(61976160,61906137,61976158,62076184,62076182);Shanghai Science and Technology Plan Project(21DZ1204800)。

摘　　要：Background External knowledge representations play an essential role in knowledge-based visual question and answering to better understand complex scenarios in the open world.Recent entity-relationship embedding approaches are deficient in representing some complex relations,resulting in a lack of topic-related knowledge and redundancy in topic-irrelevant information.Methods To this end,we propose MKEAH:Multimodal Knowledge Extraction and Accumulation on Hyperplanes.To ensure that the lengths of the feature vectors projected onto the hyperplane compare equally and to filter out sufficient topic-irrelevant information,two losses are proposed to learn the triplet representations from the complementary views:range loss and orthogonal loss.To interpret the capability of extracting topic-related knowledge,we present the Topic Similarity(TS)between topic and entity-relations.Results Experimental results demonstrate the effectiveness of hyperplane embedding for knowledge representation in knowledge-based visual question answering.Our model outperformed state-of-the-art methods by 2.12%and 3.24%on two challenging knowledge-request datasets:OK-VQA and KRVQA,respectively.Conclusions The obvious advantages of our model in TS show that using hyperplane embedding to represent multimodal knowledge can improve its ability to extract topic-related knowledge.

关 键 词：Knowledge-based visual question answering HYPERPLANE Topic-related 

分 类 号：TP391.41[自动化与计算机技术—计算机应用技术] TP391.1[自动化与计算机技术—计算机科学与技术]