基于Chinese-CLIP模型和Prompt提示机制的图文检索方法  

An Image-Text Retrieval Method Based on Chinese-CLIP Model and Prompt Mechanism

作　　者：陈道彬 张子诺 付裕彬 黎晋铭 林彬 CHEN Daobin;ZHANG Zinuo;FU Yubin;LI Jinming;LIN Bin(School of Mathematics and Statistics,Guilin University of Technology,Guilin 541006,China;Guangxi Colleges and Universities Key Laboratory of Applied Statistics,Guilin 541006,China)

机构地区：[1]桂林理工大学数学与统计学院,广西桂林541006 [2]应用统计广西高校重点实验室,广西桂林541006

出　　处：《现代信息科技》2025年第6期130-134,共5页Modern Information Technology

基　　金：广西自然科学基金项目(2019GXNSFBA245056);大学生创新创业训练计划项目(202410596733,202410596731)。

摘　　要：为提升图像文本匹配任务的准确率,提出了一种基于Chinese-CLIP模型和Prompt提示机制的图文检索方法。一方面,对文本数据进行预处理,去除停用词和标点符号后,利用BERT模型提取文本特征;另一方面,使用卷积神经网络提取图像特征,并将得到的文本与图像特征进行序列化,以实现多模态特征融合。模型训练时,先使用Chinese-CLIP大模型进行初步训练,再引入Prompt提示机制对模型进行微调。实验结果表明,所提方法在文搜图和图搜文两个任务上均有效地提升了准确率与召回率。In order to improve the accuracy of the image-text matching task,an image-text retrieval method based on Chinese-CLIP model and Prompt mechanism is proposed.On the one hand,the text data is preprocessed.After removing stop words and punctuation marks,the BERT model is used to extract text features.On the other hand,the Convolutional Neural Network is used to extract image features,and the obtained text and image features are serialized to achieve multi-modal feature fusion.When training the model,the Chinese-CLIP large model is used for preliminary training,and then the Prompt mechanism is introduced to fine-tune the model.The experimental results show that the proposed method effectively improves the accuracy and recall rate in the two tasks of text-to-image and image-to-text.

关 键 词：图文检索 多模态特征融合 Chinese-CLIP模型 Prompt提示机制 

分 类 号：TP391.4[自动化与计算机技术—计算机应用技术]