Faster Zero-shot Multi-modal Entity Linking via Visual-Linguistic Representation  

作　　者：Qiushuo Zheng Hao Wen Meng Wang Guilin Qi Chaoyu Bai 

机构地区：[1]School of Cyber Science and Engineering,Southeast University,Nanjing 211189,China [2]School of Computer Science and Engineering,Southeast University,Nanjing 211189,China [3]Key Laboratory of Computer Network and Information Integration(Southeast University),Ministry of Education,Nanjing 211189,China

出　　处：《Data Intelligence》2022年第3期493-508,共16页数据智能（英文）

摘　　要：Multi-modal entity linking plays a crucial role in a wide range of knowledge-based modal-fusion tasks, i.e., multi-modal retrieval and multi-modal event extraction. We introduce the new ZEro-shot Multi-modal Entity Linking(ZEMEL) task, the format is similar to multi-modal entity linking, but multi-modal mentions are linked to unseen entities in the knowledge graph, and the purpose of zero-shot setting is to realize robust linking in highly specialized domains. Simultaneously, the inference efficiency of existing models is low when there are many candidate entities. On this account, we propose a novel model that leverages visuallinguistic representation through the co-attentional mechanism to deal with the ZEMEL task, considering the trade-off between performance and efficiency of the model. We also build a dataset named ZEMELD for the new task, which contains multi-modal data resources collected from Wikipedia, and we annotate the entities as ground truth. Extensive experimental results on the dataset show that our proposed model is effective as it significantly improves the precision from 68.93% to 82.62% comparing with baselines in the ZEMEL task.

关 键 词：Knowledge Graph Multi-modal Learning Poly Encoders 

分 类 号：TP391.1[自动化与计算机技术—计算机应用技术] TP391.41[自动化与计算机技术—计算机科学与技术]