多模态引导的局部特征选择小样本学习方法  

Multimodal-guided Local Feature Selection for Few-shot Learning

作　　者：吕天根 洪日昌[1,2] 何军 胡社教 LÜTian-Gen;HONG Ri-Chang;HE Jun;HU She-Jiao(School of Computer Science and Information Engineering,Hefei University of Technology,Hefei 230031,China;Institute of Dataspace of Hefei Comprehensive National Science Center,Hefei 230036,China)

机构地区：[1]合肥工业大学计算机与信息学院,安徽合肥230031 [2]合肥综合性国家科学中心数据空间研究院,安徽合肥230036

出　　处：《软件学报》2023年第5期2068-2082,共15页Journal of Software

基　　金：国家自然科学基金(61932009)。

摘　　要：深度学习模型取得了令人瞩目的成绩,但其训练依赖于大量的标注样本,在标注样本匮乏的场景下模型表现不尽人意.针对这一问题,近年来以研究如何从少量样本快速学习的小样本学习被提了出来,方法主要采用元学习方式对模型进行训练,取得了不错的学习效果.但现有方法:1)通常仅基于样本的视觉特征来识别新类别,信息源较为单一;2)元学习的使用使得模型从大量相似的小样本任务中学习通用的、可迁移的知识,不可避免地导致模型特征空间趋于一般化,存在样本特征表达不充分、不准确的问题.为解决上述问题,将预训练技术和多模态学习技术引入小样本学习过程,提出基于多模态引导的局部特征选择小样本学习方法.所提方法首先在包含大量样本的已知类别上进行模型预训练,旨在提升模型的特征表达能力;而后在元学习阶段,方法利用元学习对模型进行进一步优化,旨在提升模型的迁移能力或对小样本环境的适应能力,所提方法同时基于样本的视觉特征和文本特征进行局部特征选择来提升样本特征的表达能力,以避免元学习过程中模型特征表达能力的大幅下降;最后所提方法利用选择后的样本特征进行小样本学习.在MiniImageNet、CIFAR-FS和FC-100这3个基准数据集上的实验表明,所提的小样本学习方法能够取得更好的小样本学习效果.Deep learning models have yielded impressive results in many tasks.However,the success hinges on the availability of a large number of labeled samples for model training,and deep learning models tend to perform poorly in scenarios where labeled samples are scarce.In recent years,few-shot learning(FSL)has been proposed to study how to learn quickly from a small number of samples and has achieved good performance mainly by the use of meta-learning for model training.Nevertheless,two issues exist:1)Existing FSL methods usually manage to recognize novel classes solely with the visual features of samples,without integrating information from other modalities.2)By following the paradigm of meta-learning,a model aims at learning generic and transferable knowledge from massive similar fewshot tasks,which inevitably leads to a generalized feature space and insufficient and inaccurate representation of sample features.To tackle the two issues,this study introduces pre-training and multimodal learning techniques into the FSL process and proposes a new multimodal-guided local feature selection strategy for few-shot learning.Specifically,model pre-training is first conducted on known classes with abundant samples to greatly improve the feature representation ability of the model.Then,in the meta-learning stage,the pre-trained model is further optimized by meta-learning to improve its transferability or its adaptability to the few-shot environment.Meanwhile,the local feature selection is carried out on the basis of visual features and textual features of samples to enhance the ability to represent sample features and avoid sharp degradation of the model’s representation ability.Finally,the resultant sample features are utilized for FSL.The experiments on three benchmark datasets,namely,MiniImageNet,CIFAR-FS,and FC-100,demonstrate that the proposed FSL method can achieve better results.

关 键 词：小样本学习 多模态融合 图像分类 表示学习 

分 类 号：TP18[自动化与计算机技术—控制理论与控制工程]