基于SAM的零样本多模态舌体分割方法  

Zero-Shot Multimodal Tongue Image Segmentation Based on SAM

作　　者：钟甫广 邓森耀 曾军英[1] 冯跃 钟甫东 贾旭东 

机构地区：[1]五邑大学电子与信息工程学院,广东 江门 [2]加州州立大学北岭分校计算机科学与工程学院,美国 洛杉矶

出　　处：《计算机科学与应用》2025年第3期29-38,共10页Computer Science and Application

摘　　要：舌诊通过观察舌体特征评估健康状态,而舌体分割作为智能舌诊的关键步骤,需要准确分离舌体与背景,为后续特征提取和健康分析奠定基础。然而,舌体分割目前面临着两大挑战:一是数据的稀缺性,二是现有的分割大模型(如SAM模型)对人工提示的依赖性。为了解决以上问题,本文提出了一种零样本多模态的分割方法。该方法结合SAM模型和多模态提示技术,通过两阶段框架实现:1) 初步分割和相似度聚类,利用SAM模型生成初步分割结果,并通过相似度聚类解码器筛选潜在有效分割;2) 精细化分割,利用多模态大语言模型分析舌体特征,生成精确点提示,再次输入到SAM模型中以实现高精度分割。该方法在无需特定任务训练或标注数据的情况下,实现了SAM模型在舌诊领域的智能分割应用。实验结果显示,相比于原始的SAM模型,该方法在三个舌诊数据集上的mIoU指标分别提升了27.3%,18.2%,29.7%。Tongue diagnosis assesses health status by observing tongue characteristics, and tongue segmentation, as a key step in intelligent tongue diagnosis, requires accurately separating the tongue body from the background to lay a foundation for subsequent feature extraction and health analysis. However, tongue segmentation currently faces two main challenges: data scarcity and the dependency of existing large segmentation models (such as the segment anything model) on manual prompts. To address these issues, this paper proposes a zero-shot multimodal segmentation method. This method combines the SAM model with multimodal prompt techniques and implemented in a two-stage framework: 1) initial segmentation and similarity clustering, where the SAM model generates initial segmentation results, followed by a similarity clustering decoder to filter out potentially effective segmentations;2) refined segmentation, where a multimodal large language model analyzes tongue characteristics to generate precise point prompts, which are r

关 键 词：舌体分割 零样本学习 多模态大语言模型 相似度聚类 医学图像处理 

分 类 号：TP391.4[自动化与计算机技术—计算机应用技术]