深度神经网络平均场理论综述  

Review of mean field theory for deep neural network

作　　者：颜梦玫 杨冬平 YAN Mengmei;YANG Dongping(School of Advanced Manufacturing,Fuzhou University,Quanzhou Fujian 362000,China;Quanzhou Institute of Equipment Manufacturing,Haixi Institutes,Chinese Academy of Sciences,Quanzhou Fujian 362200,China;Research Center for Human-Machine Augmented Intelligence,Zhejiang Lab,Hangzhou Zhejiang 311101,China)

机构地区：[1]福州大学先进制造学院,福建泉州362000 [2]中国科学院海西研究院泉州装备制造研究中心,福建泉州362200 [3]之江实验室混合增强智能研究中心,杭州311101

出　　处：《计算机应用》2024年第2期331-343,共13页journal of Computer Applications

基　　金：国家自然科学基金资助项目(12175242)。

摘　　要：平均场理论(MFT)为理解深度神经网络(DNN)的运行机制提供了非常深刻的见解,可以从理论上指导深度学习的工程设计。近年来,越来越多的研究人员开始投入DNN的理论研究,特别是基于MFT的一系列工作引起人们的广泛关注。为此,对深度神经网络平均场理论相关的研究内容进行综述,主要从初始化、训练过程和泛化性能这三个基本方面介绍最新的理论研究成果。在此基础上,介绍了混沌边缘和动力等距初始化的相关概念、相关特性和具体应用,分析了过参数网络以及相关等价网络的训练特性,并对不同网络架构的泛化性能进行理论分析,体现了平均场理论是理解深度神经网络机理的非常重要的基本理论方法。最后,总结了深度神经网络中初始、训练和泛化阶段的平均场理论面临的主要挑战和未来研究方向。Mean Field Theory(MFT)provides profound insights to understand the operation mechanism of Deep Neural Network(DNN),which can theoretically guide the engineering design of deep learning.In recent years,more and more researchers have started to devote themselves into the theoretical study of DNN,and in particular,a series of works based on mean field theory have attracted a lot of attention.To this end,a review of researches related to mean field theory for deep neural networks was presented to introduce the latest theoretical findings in three basic aspects:initialization,training process,and generalization performance of deep neural networks.Specifically,the concepts,properties and applications of edge of chaos and dynamical isometry for initialization were introduced,the training properties of overparameter networks and their equivalence networks were analyzed,and the generalization performance of various network architectures were theoretically analyzed,reflecting that mean field theory is a very important basic theoretical approach to understand the mechanisms of deep neural networks.Finally,the main challenges and future research directions were summarized for the investigation of mean field theory in the initialization,training and generalization phases of DNN.

关 键 词：深度神经网络 动力学 平均场理论 随机初始化 泛化性 

分 类 号：TP181[自动化与计算机技术—控制理论与控制工程]