一种进化梯度引导的强化学习算法  

Reinforcement learning algorithm guided by evolutionary gradient

作　　者：许斌[1] 练元洪 卞鸿根 刘丹 亓晋[1] XU Bin;LIAN Yuanhong;BIAN Honggen;LIU Dan;QI Jin(School of Internet of Things,Nanjing University of Posts and Telecommunications,Nanjing 210003,China)

机构地区：[1]南京邮电大学物联网学院,江苏南京210003

出　　处：《南京邮电大学学报(自然科学版)》2025年第1期99-105,共7页Journal of Nanjing University of Posts and Telecommunications:Natural Science Edition

基　　金：国家自然科学基金(62172235);江苏省高等学校基础科学(自然科学)研究项目(22KJB520028)资助项目。

摘　　要：进化算法(Evolutionary Algorithm,EA)和深度强化学习(Deep Reinforcement Learning,DRL)的组合被认为能够结合二者的优点,即EA的强大随机搜索能力和DRL的样本效率,实现更好的策略学习。然而,现有的组合方法存在EA引入所导致的策略性能不可预测性问题。提出自适应历史梯度引导机制,其利用历史梯度信息,找到平衡探索和利用的线索,从而获得较为稳定的高质量策略,进一步将此机制融合经典的进化强化学习算法,提出一种进化梯度引导的强化学习算法(Evolutionary Gradient Guided Reinforcement Learning,EGG⁃RL)。在连续控制任务方面的实验表明,EGG⁃RL的性能表现优于其他方法。The combination of evolutionary algorithms(EAs)and deep reinforcement learning(DRL)is considered to be able to combine the advantages of both,namely the strong the strong random search ability of EA and the sample efficiency of DRL,to to achieve better strategy learning.However,the existing combi⁃nation methods have the problem of unpredictability of policy performance caused by the introduction of EA.An adaptive historical gradient guidance mechanism is proposed.It uses historical gradient information to find clues to balance exploration and utilization,so as to obtain a more stable high-quality strategy.Fur⁃ther,this mechanism is integrated with the classical evolutionary reinforcement learning algorithm,and thus an evolutionary gradient guided reinforcement learning(EGG-RL)algorithm is proposed.Experi⁃ments on continuous control tasks show that the performance of EGG-RL is better than that of other methods.

关 键 词：CEM⁃RL 深度强化学习 进化算法 历史梯度 

分 类 号：TP181[自动化与计算机技术—控制理论与控制工程]