融合多模态数据的中文医学实体识别研究  

Research on Chinese Medical Named Entity Recognition with Fusion of Multimodal Data

作　　者：韩普 陈文祺 顾亮 叶东宇 景慎旗[3] Han Pu;Chen Wenqi;Gu Liang;Ye Dongyu;Jing Shenqi(School of Management,Nanjing University of Posts&Telecommunications,Jiangsu Nanjing 210003;Jiangsu Provincial Key Laboratory of Data Engineering and Knowledge Service,Jiangsu Nanjing 210023;Data Application Management Center,Jiangsu Provincial People’s Hospital,Jiangsu Nanjing 210029)

机构地区：[1]南京邮电大学管理学院,江苏南京210003 [2]江苏省数据工程与知识服务重点实验室,江苏南京210023 [3]江苏省人民医院数据应用管理中心,江苏南京210029

出　　处：《情报理论与实践》2024年第9期174-182,共9页Information Studies:Theory & Application

基　　金：国家社会科学基金项目“面向多模态医疗健康数据的知识组织模式研究”的成果,项目编号:22BTQ096。

摘　　要：[目的/意义]医学实体识别是医疗健康知识挖掘和知识组织的关键环节。深入挖掘多模态数据间语义关联可以提升医学实体识别效果,进而为领域知识补全和知识推理提供支撑。[方法/过程]提出一种基于双线性注意力融合机制的多模态中文医学实体识别模型BAF-MNER。首先通过视觉和文本编码器进行多模态医学数据的语义特征学习;接着利用双线性注意力网络实现图像和文本跨模态语义交互,并引入门控机制过滤视觉噪声;然后融合基于注意力机制的视觉特征和文本特征进而构建多模态特征表示,同时增加批量归一化层优化深度神经网络;最后将多模态特征向量输入CRF层解码获取预测标签。[结果/结论]本模型能够有效提升中文医学实体识别效果,在多模态医学数据集上的F1值较单模态基线模型提升4.07%,较多模态基线模型提升1.65%;在多模态公开数据集上的实验表明模型具有良好的泛化能力。[Purpose/significance]Medical named entity recognition is a critical step for medical and healthcare knowledge mining and knowledge organization.The semantic associations between multimodal data are mined to improve the medical entity recognition effect,which provides support for domain knowledge complementation and knowledge reasoning.[Method/process]In this paper,we propose a multimodal Chinese medical named entity recognition model BAF-MNER(Bilinear Attention Fusion-Multimodal Named Entity Recognition)based on bilinear attention fusion mechanism.The model first learns semantic features from multimodal medical data through visual and text encoders;next a bilinear attention network is utilized to achieve cross-modal semantic interaction between image and text,introducing a gating mechanism to filter the visual noise;then fusing the visual and text features based on the attention mechanism to construct the multimodal feature representations,and adding a batch normalization layer to optimize the deep neural network;finally,inputting the multimodal feature vector into a CRF layer to decode to obtain the predicted labels.[Result/conclusion]The proposed model can effectively improve the Chinese medical named entity recognition,increasing the F1 value on multimodal medical dataset by 4.07%compared with the unimodal baseline model,and 1.65%compared with the multimodal baseline model;the experiments on multimodal public dataset indicate an excellent generalization of our model.

关 键 词：多模态实体识别 多模态学习 多模态融合 残差网络 双线性注意力机制 

分 类 号：TP391.1[自动化与计算机技术—计算机应用技术] R319[自动化与计算机技术—计算机科学与技术]