基于多模态大语言模型的关系抽取研究  

Extracting Relations Based on Multimodal Large Language Model

作　　者：王震宇 朱学芳[1] 杨睿 Wang Zhenyu;Zhu Xuefang;Yang Rui(School of Information Management,Nanjing University,Nanjing 210023,China)

机构地区：[1]南京大学信息管理学院,南京210023

出　　处：《数据分析与知识发现》2025年第1期90-99,共10页Data Analysis and Knowledge Discovery

基　　金：国家社会科学基金项目(项目编号:22BTQ017)的研究成果之一。

摘　　要：【目的】利用大语言模型生成高质量辅助知识,以提升多模态关系抽取的性能。【方法】通过引入多模态相似度检测模块构建多模态提示模板,提示大语言模型将视觉信息和先验知识融入生成的高质量辅助知识中。将获得的辅助知识与原始文本输入下游文本模型,实现对实体关系的准确预测。【结果】实验表明,相较于最优基线模型,所提模型在多模态关系抽取数据集MNRE上的准确率和F1值分别提高了4.09和7.84个百分点。【局限】实验仅基于英文数据集,未考虑其他语言数据集。【结论】通过对比实验和案例分析,验证了所提模型在多模态关系抽取任务中的有效性,为未来在多模态信息抽取任务中应用大语言模型提供了方向。[Objective]This paper utilizes large language models(LLMs)to generate high-quality auxiliary knowledge,aiming to improve the performance of multimodal relation extraction.[Methods]We introduced a multimodal similarity detection module to construct multimodal prompt templates,which allow the LLM to integrate visual information and prior knowledge into the generated high-quality auxiliary knowledge.We combined the obtained auxiliary knowledge with the original text and input it into downstream text models to accurately predict entity relationships.[Results]The proposed model outperformed the best-baseline model on the MNRE dataset,achieving 4.09%and 7.84%improvements in accuracy and F1 score.[Limitations]We only examined the proposed model on English datasets.[Conclusions]Comparative experiments and case studies validate the model’s effectiveness in multimodal relation extraction.Our new model provides a direction for applying LLMs to multimodal information extraction tasks in the future.

关 键 词：关系抽取 多模态大语言模型 多模态预训练 上下文学习 

分 类 号：TP391[自动化与计算机技术—计算机应用技术]