渐进式深度集成架构搜索算法研究  

Research on Progressive Deep Ensemble Architecture Search Algorithm

作　　者：朱光辉[1,2] 祁加豪 朱振南 袁春风 黄宜华[1,2] ZHU Guang-Hui;QI Jia-Hao;ZHU Zhen-Nan;YUAN Chun-Feng;HUANG Yi-Hua(State Key Laboratory for Novel Software Technology,Nanjing University,Nanjing 210023;Department of Computer Science and Technology,Nanjing University,Nanjing 210023)

机构地区：[1]南京大学计算机软件新技术全国重点实验室,南京210023 [2]南京大学计算机科学与技术系,南京210023

出　　处：《计算机学报》2023年第10期2041-2065,共25页Chinese Journal of Computers

基　　金：国家自然科学基金项目(62102177);江苏省自然科学基金项目(BK20210181);江苏省重点研发计划项目(BE2021729);江苏省软件新技术与产业化协同创新中心资助.

摘　　要：深度神经网络已在各个领域取得巨大成功.然而,深度学习模型并不只包含深度神经网络.近年来,以深度森林为代表的深度集成学习模型,凭借着无需反向传播训练、计算开销更小、模型复杂度支持自适应确定以及表数据建模任务性能优异等特点,引起了学界和业界的广泛关注,并且取得了良好的应用效果.深度森林为探索DNN(Deep Neural Network)之外的深度学习模型打开了另一扇门.然而,现有的深度集成模型主要以深度森林为主,深度集成架构较为单一,基学习器的数量与集成方式较为固定,需要探索除深度森林之外的深度集成学习模型架构.另外,实际应用中,很难存在一种深度集成学习模型架构能够在不同数据集上均取得优异性能,尤其是对于数据特征差异较大的表格型数据集.因此,也需要一种高效的数据自适应的深度集成学习架构设计方法.为此,本文从搜索空间和搜索算法两个层面,研究提出了一种高效的基于代理模型的渐进式深度集成架构搜索方法PMPAS(Proxy Model-based Progressive Architecture Search).首先,通过归纳分析已有深度集成学习模型的特点,给出了深度集成架构的形式化定义.其次,研究提出了两种全新的深度集成架构搜索空间,即基于完全并行的搜索空间和基于有向无环图的搜索空间.然后,在上述两种搜索空间的基础上,研究提出了基于代理模型的渐进式搜索方法与算法,实现从简单到复杂逐步地在搜索空间中进行探索,并采用代理模型作为指导,降低模型评估开销.最后,本文从时间复杂度和空间复杂度两个方面对搜索算法进行分析.在分类、回归等公开的表格型数据集上的大量实验结果表明,通过PMPAS算法搜索得到的深度集成架构,其性能不仅优于已有的集成学习模型、深度学习模型以及以深度森林为代表的深度集成学习模型,而且优于已有的自动化模型选择Deep neural networks have achieved great success in various fields.However,deep learning models do not only contain deep neural networks.In recent years,due to the features such as no need for back-propagation training,lower computational overhead,support for adaptive determination of model complexity,and excellent performance in tabular data modeling tasks,the deep ensemble learning models represented by deep forest have attracted a lot of attention in the academia and industry,and achieved good application results.Deep forests open another door for exploring deep learning models beyond DNN(Deep Neural Network).However,the existing deep ensemble models are mainly based on deep forests.The deep ensemble architecture is relatively simple,and the number and integration methods of basic learners are relatively fixed.It is necessary to explore deep ensemble learning model architectures other than deep forests.In addition,in practical applications,it is difficult to have a deep ensemble learning model architecture that can achieve excellent performance on different datasets,especially for tabular datasets with large differences in data characteristics.Therefore,an efficient data-adaptive deep ensemble learning architecture design method is also needed.To this end,this paper proposes an efficient proxy model-based progressive deep ensemble architecture search method PMPAS(Proxy Model-based Progressive Architecture Search)from two levels of search space and search algorithm.First,the characteristics of existing deep ensemble learning models are analyzed by induction,and the deep ensemble architecture is formally defined.Second,the research proposes two new search spaces for deep ensemble architectures,namely a fully parallel search space and a directed acyclic graph-based search space.For each layer of the fully parallel search space,all base learners are completely independent.For the directed acyclic graph-based search space,all base learners of each cell form a directed acyclic graph.On the basis of the proposed two

关 键 词：深度学习 深度集成架构 架构搜索 代理模型 

分 类 号：TP391[自动化与计算机技术—计算机应用技术]