Derivative-free reinforcement learning:a review  

作　　者：Hong QIAN Yang YU 

机构地区：[1]National Key Laboratory for Novel Software Technology,Nanjing University,Nanjing 210023,China

出　　处：《Frontiers of Computer Science》2021年第6期75-93,共19页中国计算机科学前沿（英文版）

基　　金：This work was supported by the Program A for Outstanding PhD Candidate of Nanjing University,National Science Foundation of China(61876077);Jiangsu Science Foundation(BK20170013);Collaborative Innovation Center of Novel Software Technology and Industrialization.

摘　　要：Reinforcement learning is about learning agent models that make the best sequential decisions in unknown environments.In an unknown environment,the agent needs to explore the environment while exploiting the collected information,which usually forms a sophisticated problem to solve.Derivative-free optimization,meanwhile,is capable of solving sophisticated problems.It commonly uses a sampling-andupdating framework to iteratively improve the solution,where exploration and exploitation are also needed to be well balanced.Therefore,derivative-free optimization deals with a similar core issue as reinforcement learning,and has been introduced in reinforcement learning approaches,under the names of learning classifier systems and neuroevolution/evolutionary reinforcement learning.Although such methods have been developed for decades,recently,derivative-free reinforcement learning exhibits attracting increasing attention.However,recent survey on this topic is still lacking.In this article,we summarize methods of derivative-free reinforcement learning to date,and organize the methods in aspects including parameter updating,model selection,exploration,and parallel/distributed methods.Moreover,we discuss some current limitations and possible future directions,hoping that this article could bring more attentions to this topic and serve as a catalyst for developing novel and efficient approaches.

关 键 词：reinforcement learning derivative-free optimization neuroevolution reinforcement learning neural architecture search 

分 类 号：TP181[自动化与计算机技术—控制理论与控制工程]