CLIP-SP:Vision-language model with adaptive prompting for scene parsing  

作　　者：Jiaao Li Yixiang Huang Ming Wu Bin Zhang Xu Ji Chuang Zhang 

机构地区：[1]School of Artificial Intelligence,Beijing University of Posts and Telecommunications,Beijing 100876,China

出　　处：《Computational Visual Media》2024年第4期741-752,共12页计算可视媒体（英文版）

摘　　要：We present a novel framework,CLIPSP,and a novel adaptive prompt method to leverage pre-trained knowledge from CLIP for scene parsing.Our approach addresses the limitations of DenseCLIP,which demonstrates the superior image segmentation provided by CLIP pre-trained models over ImageNet pre-trained models,but struggles with rough pixel-text score maps for complex scene parsing.We argue that,as they contain all textual information in a dataset,the pixel-text score maps,i.e.,dense prompts,are inevitably mixed with noise.To overcome this challenge,we propose a two-step method.Firstly,we extract visual and language features and perform multi-label classification to identify the most likely categories in the input images.Secondly,based on the top-k categories and confidence scores,our method generates scene tokens which can be treated as adaptive prompts for implicit modeling of scenes,and incorporates them into the visual features fed into the decoder for segmentation.Our method imposes a constraint on prompts and suppresses the probability of irrelevant categories appearing in the scene parsing results.Our method achieves competitive performance,limited by the available visual-language pre-trained models.Our CLIP-SP performs 1.14%better(in terms of mIoU)than DenseCLIP on ADE20K,using a ResNet-50 backbone.

关 键 词：visual-language pre-trained model scene parsing adaptive prompt 

分 类 号：TP391.41[自动化与计算机技术—计算机应用技术]