稀疏奖励环境中的分层强化学习  

Hierarchical Reinforcement Learning in a Sparse Reward Environment

作　　者：欧阳名三[1] 冯舒心 OUYANG Mingsan;FENG Shuxin(College of Electrical and Information Engineering,Anhui University of Science and Technology,Huainan Anhui 232001,China)

机构地区：[1]安徽理工大学电气与信息工程学院,安徽淮南232001

出　　处：《佳木斯大学学报（自然科学版）》2022年第2期54-57,共4页Journal of Jiamusi University：Natural Science Edition

基　　金：国家自然科学基金项目(51874010);安徽省重点研究与开发计划(202004a05020080)。

摘　　要：针对强化学习的稀疏奖励问题,提出一种基于子目标的分层强化学习算法。该算法利用分层学习框架将学习任务分解为两个抽象级别,上层控制器根据策略选择子目标,底层控制器中的批判函数提供适当的内部奖励,使用神经网络训练价值函数,进行动作的选择和策略的更新,直到完成子目标或完成整个学习任务,子目标集合由启发式异常检测结合密度峰值聚类自动创建。与传统强化学习算法以及基于k-means的分层强化学习算法在路径导航仿真问题中进行了对比实验,实验表明,该算法在稀疏延时反馈的环境中具有较高的稳定性和学习效率。Aiming at the sparse reward problem of reinforcement learning,a hierarchical reinforcement learning algorithm based on subgoals.which uses a hierarchical learning framework to decompose the learning task into two abstract levels.The upper controller selects subgoals according to policy,and the critical function in the bottom controller provides appropriate internal rewards,then use neural networks to train the value function,selects actions and updates the policy until subgoals are completed or learning task is completed.The set of sub-goals is automatically created by rewarding abnormal point detection combined with density peak clustering.Compared with the traditional reinforcement learning algorithm and the hierarchical reinforcement learning algorithm based on k-means in the path navigation simulation problem,the experiment shows that the algorithm has high stability and learning efficiency in the environment of sparse delayed feedback.

关 键 词：分层强化学习 稀疏奖励 密度峰值聚类 路径导航 

分 类 号：TP181[自动化与计算机技术—控制理论与控制工程]