基于距离置信度分数的多模态融合分类网络  

Multimodal Fusion Classification Network Based on Distance Confidence Score

作　　者：郑德重 杨媛媛[1] 黄浩哲[3] 谢哲 李文涛[3] ZHENG Dezhong;YANG Yuanyuan;HUANG Haozhe;XIE Zhe;LI Wentao(Laboratory for Medical Imaging Informatics,Shanghai Institute of Technical Physics,Chinese Academy of Sciences,Shanghai 200080,China;University of Chinese Academy of Sciences,Beijing 100049,China;Fudan University Shanghai Cancer Center,Shanghai 200032,China)

机构地区：[1]中国科学院上海技术物理研究所医学影像信息学实验室,上海200080 [2]中国科学院大学,北京100049 [3]复旦大学附属肿瘤医院,上海200032

出　　处：《上海交通大学学报》2022年第1期89-100,共12页Journal of Shanghai Jiaotong University

基　　金：人工智能医学信息系统软件临床试验技术规范资助项目(2019YFC0118805)。

摘　　要：使用多模态数据建模可以有效地克服单一模态信息量不足的问题,大大提高模型的性能.但在量化神经网络模型置信度,尤其是对于多模态融合模型方面并没有很多进展.基于此,提出一种基于嵌入的方法,在嵌入空间中通过计算样本间的距离进行局部密度估计,进而计算模型的置信度分数.该方法具备可扩展性,不仅可以用于单一模态模型,还可以用于多模态融合模型置信度的度量.此外,所提方法还可以用来评估和量化不同模态数据对于多模态融合模型的影响程度.Multimodal data modeling can effectively overcome the problem of insufficient information in a single mode and can greatly improve the performance of model.However,not much progress has been made in quantifying the confidence of neural network models,especially for multimodal fusion models.This paper proposes a method based on embedding,which calculates the local density estimation in the embedding space by calculating the distance between samples,and then calculates the confidence score of the model.The proposed method is scalable and can be used not only for a single modal model,but also for the confidence measurement of multimodal fusion model.In addition,it can also evaluate and quantify the influences of different modal data on the multimodal fusion model.

关 键 词：置信度 多模态融合 神经网络 

分 类 号：TP181[自动化与计算机技术—控制理论与控制工程]