强化学习中动态ε的贪婪探索策略  

A dynamicε-Greedy exploration strategy in reinforcement learning

作　　者：孔燕 曹俊豪 杨智超 芮烨锋 KONG Yan;CAO Jun-hao;YANG Zhi-chao;RUI Ye-feng(School of Computer Science,Nanjing University of Information Science and Technology,Nanjing 210044,China;Engineering Research Center of Digital Forensics,Ministry of Education,Nanjing University of Information Science and Technology,Nanjing 210044,China)

机构地区：[1]南京信息工程大学计算机学院,南京210044 [2]南京信息工程大学数字取证教育部工程研究中心,南京210044

出　　处：《信息技术》2024年第7期60-64,70,共6页Information Technology

基　　金：国家自然科学基金(61602254)。

摘　　要：随着强化学习领域的成熟,ε-贪婪方法被广泛运用在强化学习中,例如深度Q网络。但是,对于ε-贪婪方法每次选择动作,它有一定概率选择非最优的动作,导致不断探索。在此背景下,提出了一种动态ε-贪婪方法(DEG)和Dueling Actor-Critic框架(ACDD),能够平衡强化学习中的探索和利用问题。DEG将状态输入到ACDD框架得到优势值来自动调整ε的值,从而保持探索和利用之间的更好平衡。该实验在多臂老虎机任务中对DEG进行测试,将累计平均奖励和最优动作选择率作为评估标准。与一些广泛使用的方法相比,DEG可以达到更高的平均累积奖励和最优动作选择率,并提高了性能。As the field of RL matures,ε-greedy method is widely used in RL,such as deep Q network.However,ε-greedy has a certain probability to choose other actions that are not optimal when choosing actions,leading to constant exploration.In this context,a Dynamicε-greedy algorithm(DEG)and Dueling Actor-Critic framework(ACDD)are proposed,which are able to balance exploration and exploitation problems in RL.DEG inputs the state into the ACDD framework to get the advantage value to automatically adjust the value ofε,thus maintaining a better balance between exploration and exploitation.This experiment tests the performance of DEG in a Multi-Armed Bandit task,using the average cumulative reward and the optimal action selection rate as evaluation criteria.Compared with some widely used algorithms,DEG can achieve higher average cumulative reward and optimal action selection rate,and improve performance.

关 键 词：深度强化学习 探索和利用 动态化ε Dueling Actor-Critic框架 多臂老虎机 

分 类 号：TP18[自动化与计算机技术—控制理论与控制工程]