基于知识蒸馏的高效生物医学命名实体识别模型  

Faster biomedical named entity recognition based on knowledge distillation

作　　者：胡滨 耿天玉 邓赓 段磊[1] HU Bin;GENG Tianyu;DENG Geng;DUAN Lei(School of Computer Science,Sichuan University,Chengdu 610065,China)

机构地区：[1]四川大学计算机学院,成都610065

出　　处：《清华大学学报（自然科学版）》2021年第9期936-942,共7页Journal of Tsinghua University(Science and Technology)

基　　金：国家自然科学基金资助项目(61906126,61972268,61572332)。

摘　　要：在生物医学文献挖掘领域中,已有的BioBert采用预训练的方式在生物医学命名实体识别(BioNER)任务中表现出优秀的实体识别性能,却存在模型规模过大、速度过慢的缺点。针对BioBert网络模型如何高效压缩问题,该文提出了一种高效生物医学名称实体识别模型,命名为FastBioNER模型。该模型采用动态知识蒸馏方式对BioBert模型进行压缩,首先,通过动态权重函数模拟真实的学习行为,即在模型训练过程中动态调整各部分损失函数的重要程度;其次,采用动态知识蒸馏方式将训练后的BioBert作为教师模型,压缩到一个规模较小的学生模型中;最后,对提出的FastBioNER模型在已公开数据集NCBI疾病、BC5CDR-chem和BC4CHEMD上进行了实验验证。实验结果表明:提出的FastBioNER模型在3个数据集中获得除BioBert外最高F1值分别为88.63%、92.82%和92.60%,并分别以损失1.10%、0.86%、0.15%的F1值为代价,将BioBert的模型大小压缩了39.26%,同时推理时间缩短了46.17%。Many biomedical literature mining systems use the pre-training language model BioBert which provides state-of-the-art biomedical named entity recognition after pre-training.However,BioBert is too large scale and slow.This paper presents a faster biomedical named entity recognition model,FastBioNER,that is based on knowledge distillation.FastBioNER compresses the BioBert model using dynamic knowledge distillation.A dynamic weight function is used to simulate the real learning behavior to adjust the importance of the loss function of each part during training.Then,the trained BioBert is compressed into a small student model by dynamic knowledge distillation.The FastBioNER model was validated on three common data sets,NCBI disease,BC5CDR-chem and BC4CHEMD.The tests show that FastBioNER had the highest F1 values after BioBert at 88.63%,92.82% and 92.60% for the three data sets while reducing the BioBert model size by 39.26%and the inference time by 46.17%at the cost of 1.10%,0.86% and 0.15% smaller F1.

关 键 词：自然语言处理 生物医学信息学 命名实体识别 知识蒸馏 

分 类 号：TP391.1[自动化与计算机技术—计算机应用技术]