多模态信息抽取研究综述  

Survey on Multimodal Information Extraction Research

作　　者：王永胜 李培峰[1] 王中卿[1] 朱巧明[1] WANG Yong-Sheng;LI Pei-Feng;WANG Zhong-Qing;ZHU Qiao-Ming(School of Computer Science and Technology,Soochow University,Suzhou 215006,China)

机构地区：[1]苏州大学计算机科学与技术学院,江苏苏州215006

出　　处：《软件学报》2025年第4期1665-1691,共27页Journal of Software

基　　金：国家自然科学基金(62276177,61836007);江苏高校优势学科建设工程项目。

摘　　要：多模态信息抽取任务是指从非结构化或半结构化的多模态数据(包含文本和图像等)中提取结构化知识.其研究内容主要包含多模态命名实体识别、多模态实体关系抽取和多模态事件抽取.首先对多模态信息抽取任务进行分析,然后对多模态命名实体识别、多模态实体关系抽取和多模态事件抽取这3个子任务的共同部分,即多模态表示和融合模块进行归纳和总结.随后梳理上述3个子任务的常用数据集和主流研究方法.最后总结多模态信息抽取的研究趋势并分析该研究存在的问题和挑战,为后续相关研究提供参考.Multimodal information extraction is a task to extract structured knowledge from unstructured or semi-structured multimodal data(such as text and images).It includes multimodal named entity recognition,multimodal relation extraction,and multimodal event extraction.This study analyzes multimodal information extraction tasks and summarizes the common part of the above three subtasks,i.e.,a multimodal representation and fusion module.Moreover,it sorts out the commonly used datasets and mainstream research methods of the above three subtasks.Finally,it outlines research trends in multimodal information extraction and analyzes the existing problems and challenges in this field to provide a reference for future research.

关 键 词：多模态信息抽取 多模态命名实体识别 多模态实体关系抽取 

分 类 号：TP18[自动化与计算机技术—控制理论与控制工程]