深度学习经验损失函数地貌分析研究进展  

Empirical loss landscape analysis in deep learning:A survey

作　　者：梁若冰 刘波[1,2] 孙越泓 LIANG Ruobing;LIU Bo;SUN Yuehong(Academy of Mathematics and Systems Science,Chinese Academy of Sciences,Beijing 100190,China;National Center for Mathematics and Interdisciplinary Sciences,Beijing 100190,China;School of Mathematical Sciences,University of Chinese Academy of Sciences,Beijing 100049,China;School of Mathematical Sciences,Nanjing Normal University,Nanjing 210046,China)

机构地区：[1]中国科学院数学与系统科学研究院,北京100190 [2]国家数学与交叉科学研究中心,北京100190 [3]中国科学院大学数学科学学院,北京100049 [4]南京师范大学数学科学学院,南京210046

出　　处：《系统工程理论与实践》2023年第3期813-823,共11页Systems Engineering-Theory & Practice

基　　金：中国科学院前沿科学重点研究计划(QYZDB-SSW-SYS020)。

摘　　要：经验损失函数地貌分析是揭示深度学习优化问题易优性本质的有效途径,已成为机器学习及数学优化领域的重要研究方向.本文回顾了经验损失函数地貌分析的研究进展和挑战,主要包括局部极小点的数量与空间分布、最优点之间的连通性、临界点的最优性、梯度下降法收敛性、经验损失函数地貌可视化等.Empirical loss landscape analysis is critical to reveal reasons why deep networks are easily optimizable,and has aroused considerable interests in machine learning and mathematical optimization.The main goal of this investigation is to provide a comprehensive state-of-the-art review of the empirical loss landscape analysis,including number and spatial distribution of local minima,connectivity between global optima,global optimality of critical points,convergence of gradient descent,and visualization of empirical loss landscape.This review also identifies challenges and highlights opportunities for future work.

关 键 词：深度学习 经验损失函数 地貌分析 

分 类 号：TP18[自动化与计算机技术—控制理论与控制工程]