融合计算机视觉和结构化数据的多模态模型在糖尿病视网膜病变转诊中的应用  

Application of multimodal model in diabetic retinopathy referral based on computer vision and structured data

作　　者：赵雅丽[1] 金雪梅 肖海祥 王莹[2] ZHAO Yali;JIN Xuemei;XIAO Haixiang;WANG Ying(Department of Ophthalmology,Lixiang Eye Hospital of Soochow University,Suzhou 215000,Jiangsu Province,China;Department of Ophthalmology,Suzhou Municipal Hospital)

机构地区：[1]苏州大学理想眼科医院眼科,江苏苏州215000 [2]苏州市立医院眼科

出　　处：《中国数字医学》2024年第7期29-35,共7页China Digital Medicine

基　　金：苏州市科技计划项目(SKY2021025)。

摘　　要：目的:融合计算机视觉和临床结构化数据,探讨多模态模型在糖尿病视网膜病变(DR)转诊中的应用。方法:纳入苏州大学理想眼科医院和苏州市立医院就诊的糖尿病患者,收集眼底照片及临床资料,根据临床指南相关标准,将患者分为DR无需转诊组和需转诊组。以EfficientNetV2S作为后骨框架进行迁移学习,构建计算机视觉模型α;并融合计算机视觉模型α的输出以及患者临床结构化数据,使用H2O的AutoML平台,建立多模态模型β,判断糖尿病患者是否需要DR转诊。结果:视觉分类模型α在内部验证集中,准确度为0.918,敏感度为1.000,特异度为0.891,ROC曲线下面积(AUC)为0.946;在外部测试集中,准确度为0.879,敏感度为1.000,特异度为0.831,AUC为0.918。基于XGBoost算法的多模态模型β在内部验证集准确度为0.965,特异度为0.953,敏感度为1.000,AUC为0.977;在外部测试集中,其准确度为0.985,特异度为0.986,敏感度为0.983,AUC为0.984。SHAP特征可视化结果,可以观察到基于XGBoost算法的最佳模型中,排名靠前的变量在二分类结局患者中的分布。其中,空腹血糖、胰岛素、谷草转氨酶、收缩压及三酰甘油与DR转诊呈正相关,而高密度脂蛋白与DR转诊呈负相关。结论:相比单模态模型,基于计算机视觉和结构化数据的多模态融合模型判断DR患者转诊的准确性显著提高。Objective To explore the application of multimodal model in diabetic retinopathy(DR)referral by integrating computer vision(CV)and structured data.Methods Diabetic patients from Lixiang Eye Hospital of Soochow University and Suzhou Municipal Hospital were enrolled.Fundus photographs and clinical data were collected,and patients were divided into the DR group without referral and the DR group requiring referral according to the relevant criteria of clinical guidelines.EfficientNetV2S was used as the posterior bone framework for transfer learning,and the CV model α was constructed.And then integrated with the output of the CV modelαand the patient's clinical structured data,the multimodal model β was established by using H2O's AutoML platform to determine whether diabetic patients need DR referral or not.Results In the validation dataset,the accuracy of the model α was 0.918,the sensitivity was 1.000,the specificity was 0.891 and the area under ROC curve(AUC)was 0.946.In the external test dataset,the accuracy was 0.879,the sensitivity was 1.000,the specificity was 0.831 and the AUC was 0.918.In the internal validation set,the accuracy of the multimodal model β was 0.965,the specificity was 0.953,the sensitivity was 1.000,and the AUC was 0.977.In the external test set,the accuracy was 0.985,the specificity was 0.986,the sensitivity was 0.983,and the AUC was 0.984.Furthermore,the XGBoost model results were visualized by SHapley Additive exPlanation,which can observe the distribution of the top variables in patients with binary outcomes in the optimal model based on the XGBoost algorithm.Fasting blood glucose,insulin,AST,systolic pressure and triacylglycerol were positively correlated with DR referral,while high-density lipoprotein cholesterol was negatively correlated with DR referral.Conclusion Compared with the single modal model,the multimodal integration model based on CV and structured clinical data can significantly improve the accuracy of DR patients referral.

关 键 词：多模态融合 糖尿病视网膜病变 机器学习 计算机视觉 人工智能 

分 类 号：R197.3[医药卫生—卫生事业管理] R319[医药卫生—公共卫生与预防医学]