基于权重值的竞争深度双Q网络算法  

Weighted dueling double deep Q-network

作　　者：汪晨曦 赵学艳[1] 郭新[2] WANG Chenxi;ZHAO Xueyan;GUO Xin(School of Automation Science and Engineering,South China University of Technology,Guangzhou 510641;School of Mechanical and Electrical Engineering,Guangdong Communication Polytechnic,Guangzhou 510650)

机构地区：[1]华南理工大学自动化科学与工程学院,广州510641 [2]广东交通职业技术学院,机电工程学院,广州510650

出　　处：《南京信息工程大学学报（自然科学版）》2021年第5期564-570,共7页Journal of Nanjing University of Information Science & Technology（Natural Science Edition）

基　　金：国家自然科学基金(61873099,62073144);广东省自然科学基金(2020A1515010441);广州市科技计划(202002030158,202002030389);广东省青年创新人才项目(2018GkQNCX005)。

摘　　要：在深度强化学习中,深度Q网络算法存在严重高估动作值问题,使得智能体的表现不尽人意.尽管深度双Q网络和竞争网络结构可以部分缓解高估带来的影响,但引入双Q网络的同时,有时也会低估动作值.本文提出了一种基于权重值的竞争深度双Q网络算法(Weighted Dueling Double Deep Q-Network,WD3QN),把改进的双估计器及竞争网络结构结合至深度Q网络中,将学习到的可能动作值进行加权产生最终动作值,有效减少估计误差.最后,将算法应用于Open AI Gym平台上的CartPole经典控制问题,仿真结果显示:与已有算法对比,本算法有更好的学习效果,收敛性和训练速度均有提升.In deep reinforcement learning,the deep Q-network algorithm seriously overestimates the action value,which degrades the performance of agents.The double deep Q-network and dueling network structure can partially alleviate the impact of overestimation,sometimes the former one even underestimate the action value.Here,a Weighted Dueling Double Deep Q-Network(WD3QN)algorithm is proposed,in which the improved double estimators and dueling network structure are combined into the deep Q-network,and the learned possible action values are weighted to produce the final action value,which can effectively reduce the estimation error.Finally,the algorithm is applied to the classical CartPole control problem on the open AI Gym platform.The simulation results show that compared with other existing algorithms,the proposed algorithm has better learning effect,convergence and training speed.

关 键 词：深度强化学习 深度双Q网络 竞争网络结构 权重值 

分 类 号：TP273[自动化与计算机技术—检测技术与自动化装置]