安全强化学习及其在机器人系统中的应用综述  

Safe reinforcement learning and its applications in robotics:A survey

作　　者：张昌昕 张兴龙 徐昕[1] 陆阳 ZHANG Chang-xin;ZHANG Xing-long;XU Xin;LU Yang(College of Intelligence Science and Technology,National University of Defense Technology,Changsha Hunan 410000,China)

机构地区：[1]国防科技大学智能科学学院,湖南长沙410000

出　　处：《控制理论与应用》2023年第12期2090-2103,共14页Control Theory & Applications

基　　金：国家自然科学基金项目(62003361,U21A20518)资助。

摘　　要：强化学习是一类通过与环境交互实现序贯优化决策的机器学习方法,已经在游戏、推荐系统及自然语言处理等任务中得到了应用.然而,强化学习算法应用于真实世界中的机器人系统时,如何保证安全性仍然面临挑战.近年来,针对机器人系统的安全强化学习方法研究已经成为热点方向,获得了机器人和强化学习领域的广泛关注.本文结合现有的工作,综述了安全强化学习理论和方法的重要成果和发展趋势,并重点关注了现有方法在机器人领域的适用性.本文首先给出了安全强化学习的一般问题描述.其次,从方法和性能的角度重点介绍了该领域的最新重要进展,包括约束策略优化、控制障碍函数、安全过滤器和对抗性博弈训练等方法,以及安全强化学习方法在地面移动机器人系统、无人飞行器和其他机器人系统中的应用情况.最后,对该领域的未来研究方向进行了展望和探讨.Reinforcement learning is a kind of machine learning method that realizes sequential optimization decisions by interacting with the environment.It has been applied in games,recommendation systems and natural language processing.However,it is still a challenge to ensure the safety of reinforcement learning algorithms when applied to robotics in the real world.In recent years,the safe reinforcement learning methods for robotics systems have become a hot research direction,gaining extensive attention in robotics and reinforcement learning communities.This paper surveys important achievements and development tendency of safe reinforcement learning based on the existing work and focuses on their applicability in robotics.This paper first introduces the general problem description of safe reinforcement learning.Then we focus on the latest significant progress in this field from the perspective of method and performance,including constraint policy optimization,control barrier function,safety filter and adversarial training methods,and their applications in autonomous driving vehicles,unmanned aerial vehicles and other robotic systems.Finally,the future research direction of this field is prospected and discussed.

关 键 词：机器人 安全强化学习 约束马尔可夫决策过程 鲁棒性 

分 类 号：TP242[自动化与计算机技术—检测技术与自动化装置] TP18[自动化与计算机技术—控制科学与工程]