基于伪孪生网络双层优化的对比学习  

Contrastive Learning Based on Bilevel Optimization of Pseudo Siamese Networks

作　　者：陈庆宇 季繁繁 袁晓彤[2,3,4] CHEN Qingyu;JI Fanfan;YUAN Xiaotong(School of Automation,Nanjing University of Information Science and Technology,Nanjing 210044;Engineering Research Center of Digital Forensics Ministry of Education,Nanjing University of Information Science and Technology,Nanjing 210044;Jiangsu Key Laboratory of Big Data Analysis Technology,Nanjing University of Information Science and Technology,Nanjing 210044;School of Computer Science,Nanjing University of Information Science and Technology,Nanjing 210044)

机构地区：[1]南京信息工程大学自动化学院,南京210044 [2]南京信息工程大学数字取证教育部工程研究中心,南京210044 [3]南京信息工程大学江苏省大数据分析技术重点实验室,南京210044 [4]南京信息工程大学计算机学院,南京210044

出　　处：《模式识别与人工智能》2022年第10期928-938,共11页Pattern Recognition and Artificial Intelligence

基　　金：科技创新2030-“新一代人工智能”重大项目(No.2018AAA0100400);国家自然科学基金项目(No.U21B2049,61876090,61936005)资助。

摘　　要：目前,基于伪孪生网络的对比学习算法使用各种组件以获得最优学生网络,但忽略教师网络在下游任务中的表现,因此,文中提出基于伪孪生网络双层优化的对比学习,促进学生网络和教师网络相互学习,获得最优教师网络。双层优化策略包括基于近邻优化的学生网络优化策略和基于随机梯度下降的教师网络优化策略。基于近邻优化的学生网络优化策略让教师网络成为约束项,帮助学生网络更好地向教师网络学习。基于随机梯度下降的教师网络优化策略求解近似教师网络,梯度更新教师网络。在5个数据集上的实验表明,文中算法取得较高的k-NN(k=1)分类精度和线性分类精度,特别在批次大小较小时,优势较大。At present,various designs are applied in contrastive learning algorithms based on pseudo siamese networks to acquire the best student network.However,the performance of teacher network in downstream tasks is ignored.Therefore,an algorithm of contrastive learning based on bilevel optimization of pseudo siamese networks(CLBO)is proposed to acquire the best teacher network by promoting the learning between student and teacher networks.The bilevel optimization strategy includes student network optimization strategy based on nearest neighbor optimization and teacher network optimization strategy based on stochastic gradient descent.The teacher network is regarded as a constraint term through the student network optimization strategy based on nearest neighbor optimization to help the student network learn better from the teacher network.The parameters are calculated by the teacher network optimization strategy based on stochastic gradient descent to update the teacher network.Experiments on 5 datasets show that CLBO performs better than other algorithms in k-NN classification and linear classification tasks.Especially,the advantages of CLBO is obvious when the batch size is smaller.

关 键 词：对比学习 双层优化 学生网络 近邻优化 教师网络 随机梯度下降 

分 类 号：TP391[自动化与计算机技术—计算机应用技术]