Clustered Reinforcement Learning  

作　　者：Xiao MA Shen-Yi ZHAO Zhao-Heng YIN Wu-Jun LI 

机构地区：[1]National Key Laboratory for Novel Software Technology,Department of Computer Science and Technology,Nanjing University,Nanjing 210023,China [2]Department of Electrical Engineering and Computer Sciences,University of California,Berkeley,CA 94720-1770,USA

出　　处：《Frontiers of Computer Science》2025年第4期43-57,共15页计算机科学前沿(英文版)

基　　金：supported by the National Natural Science Foundation of China(Gtant No.62192783);Fundamental Research Funds for the Central Universities(No.020214380108).

摘　　要：Exploration strategy design is a challenging problem in reinforcement learning(RL),especially when the environment contains a large state space or sparse rewards.During exploration,the agent tries to discover unexplored(novel)areas or high reward(quality)areas.Most existing methods perform exploration by only utilizing the novelty of states.The novelty and quality in the neighboring area of the current state have not been well utilized to simultaneously guide the agent’s exploration.To address this problem,this paper proposes a novel RL framework,called clustered reinforcement learning(CRL),for efficient exploration in RL.CRL adopts clustering to divide the collected states into several clusters,based on which a bonus reward reflecting both novelty and quality in the neighboring area(cluster)of the current state is given to the agent.CRL leverages these bonus rewards to guide the agent to perform efficient exploration.Moreover,CRL can be combined with existing exploration strategies to improve their performance,as the bonus rewards employed by these existing exploration strategies solely capture the novelty of states.Experiments on four continuous control tasks and six hard-exploration Atari-2600 games show that our method can outperform other state-of-the-art methods to achieve the best performance.

关 键 词：deep reinforcement learning EXPLORATION count-based method CLUSTERING K-MEANS 

分 类 号：TP181[自动化与计算机技术—控制理论与控制工程]