基于Wasserstein距离的在线机器学习算法研究  

Online machine learning algorithms based on Wasserstein distance

作　　者：李兆恩 张智海[1] LI ZhaoEn;ZHANG Zhi-Hai(Department of Industrial Engineering,Tsinghua University,Beijing 100084,China)

机构地区：[1]清华大学工业工程系,北京100084

出　　处：《中国科学：技术科学》2023年第7期1031-1042,共12页Scientia Sinica(Technologica)

基　　金：国家自然科学基金联合基金专家库构建方法及策略研究项目(编号:J2124091)资助。

摘　　要：本文主要研究基于Wasserstein距离的在线机器学习算法,并分别针对分类和回归问题,提出两个鲁棒的在线学习算法.本文首先在特征-标签空间中对Wasserstein距离进行变形,得到了易于处理和计算的变形式.进而,将在线梯度下降(online gradient descent,OGD)算法和Wasserstein距离变形式结合,分别针对在线分类问题和在线回归问题提出了两种具有较好鲁棒性的在线机器学习算法.文章对提出的新算法累积误差值(Regret,后面用Regret指代)进行了分析,证明了算法的Regret与训练轮次T满足O(√T)关系.算法的收敛性可基于算法Regret分析得到,可证明在算法训练轮次T趋于无穷时,算法训练出的模型会收敛到理论最优模型.最后,将所提出算法与FTRL(follow-the-regularized-leader)算法、OGD算法、采用批量学习训练方法的机器学习算法进行数值实验对比.在模拟数据集和真实数据集的实验中,所提出在线学习算法准确率、鲁棒性和模型泛化性能均优于FTRL算法和OGD算法;针对大规模数据集时,虽然所提出的在线学习算法准确率与批量学习相关算法相近,但在线学习算法可以大量节省每次输入新数据后的训练时间和数据存储空间;在数据量较小且数据质量较差时,在线学习算法鲁棒性和模型泛化性能表现更优,甚至优于批量学习相关算法.This paper analyzes an online learning method based on Wasserstein distance and proposes two robust online learning algorithms for solving classification and regression problems.The Wasserstein distance is reformulated into a tractable form in a feature-label space.Then,based on the online gradient descent(OGD)algorithm and Wasserstein distance reformulation,two robust online learning algorithms are proposed for solving online classification and regression problems.This paper analyzes the regrets of the two proposed algorithms and proves the O(√T)regret,where T denotes training rounds.According to this regret,we can prove that when T tends to infinity,new algorithms can converge to optimal models.Some experiments are conducted in which the proposed algorithms are compared with the follow-the-regularized-leader,OGD,and related batch-learning algorithms.When simulated and real-world datasets are used,the proposed online learning algorithms outperform the other abovementioned online learning algorithms in terms of prediction accuracy,robustness,and model generalization.When the size of the dataset is large enough,the new algorithms exhibit a comparable prediction accuracy to the batch learning algorithms and better robustness and model generalization.Hence,the proposed algorithms save training time and data storage space.When the dataset is of poor quality,the proposed algorithms exhibit a higher prediction accuracy than the batch learning algorithms.

关 键 词：Wasserstein距离 在线机器学习 在线梯度下降算法 算法累积误差值 

分 类 号：TP181[自动化与计算机技术—控制理论与控制工程]