基于随机加权三重Q学习的异策略最大熵强化学习算法  

Off-policy Maximum Entropy Deep Reinforcement Learning Algorithm Based on Randomly Weighted Triple Q-Learning

作　　者：范静宇 刘全[1,2,3,4] FAN Jing-yu;LIU Quan(School of Computer Science and Technology,Soochow University,Suzhou,Jiangsu 215006,China;Provincial Key Laboratory for Computer Information Processing Technology,Soochow University,Suzhou,Jiangsu 215006,China;Key Laboratory of Symbol Computation and Knowledge Engineering of Ministry of Education,Jilin University,Changchun 130012,China;Collaborative Innovation Center of Novel Software Technology and Industrialization,Nanjing 210000,China)

机构地区：[1]苏州大学计算机科学与技术学院,江苏苏州215006 [2]苏州大学江苏省计算机信息处理技术重点实验室,江苏苏州215006 [3]吉林大学符号计算与知识工程教育部重点实验室,长春130012 [4]软件新技术与产业化协同创新中心,南京210000

出　　处：《计算机科学》2022年第6期335-341,共7页Computer Science

基　　金：国家自然科学基金(61772355,61702055,61502323,61502329);江苏省高等学校自然科学研究重大项目(18KJA520011,17KJA520004);吉林大学符号计算与知识工程教育部重点实验室资助项目(93K172014K04,93K172017K18);苏州市应用基础研究计划工业部分(SYG201422);江苏省高校优势学科建设工程资助项目。

摘　　要：强化学习是机器学习中一个重要的分支,随着深度学习的发展,深度强化学习逐渐发展为强化学习研究的重点。因应用广泛且实用性较强,面向连续控制问题的无模型异策略深度强化学习算法备受关注。同基于离散动作的Q学习一样,类行动者-评论家算法会受到动作值高估问题的影响。在类行动者-评论家算法的学习过程中,剪切双Q学习可以在一定程度上解决动作值高估的问题,但同时也引入了一定程度的低估问题。为了进一步解决类行动者-评论家算法中的高低估问题,提出了一种新的随机加权三重Q学习方法。该方法可以更好地解决类行动者-评论家算法中的高低估问题。此外,将这种新的方法与软行动者-评论家算法结合,提出了一种新的基于随机加权三重Q学习的软行动者-评论家算法,该算法在限制Q估计值在真实Q值附近的同时,通过随机加权方法增加Q估计值的随机性,从而有效解决了学习过程中对动作值的高低估问题。实验结果表明,相比SAC算法、DDPG算法、PPO算法与TD3算法等深度强化学习算法,SAC-RWTQ算法可以在gym仿真平台中的多个Mujoco任务上获得更好的表现。Reinforcement learning is an important branch of machine learning.With the development of deep learning,deep reinforcement learning research has gradually developed into the focus of reinforcement learning research.Model-free off-policy deep reinforcement learning algorithms for continuous control attract everyone’s attention because of their strong practicality.Like Q-learning,algorithms based on actor-critic suffer from the problem of overestimations.To a certain extent,clipped double Q-lear-ning method solves the effect of the overestimation in actor-critic algorithms,but it also introduces underestimation to the lear-ning process.In order to further solve the problems of overestimation and underestimation in the actor-critic algorithms,a new learning method,randomly weighted triple Q-learning method is proposed.In addition,combining the new method with the soft actor critic algorithm,a new soft actor critic algorithm based on randomly weighted triple Q-learning is proposed.This algorithm not only limits the Q estimation value near the real Q value,but also increases the randomness of the Q estimation value through randomly weighted method,so as to solve the problems of overestimation and underestimation of action value in the learning process.Experiment results show that,compared to the SAC algorithm and other currently popular deep reinforcement learning algorithms such as DDPG,PPO and TD3,the SAC-RWTQ algorithm has better performance on several Mujoco tasks on the gym simulation platform.

关 键 词：Q学习 深度学习 异策略强化学习 连续动作空间 最大熵 软行动者—评论家算法 

分 类 号：TP181[自动化与计算机技术—控制理论与控制工程]