基于离线强化学习的研究综述  

Survey of Research on Offline Reinforcement Learning

作　　者：陈锶奇 耿婕 汪云飞 余伟驰 赵佳宁 王仕超 CHEN Siqi;GENG Jie;WANG Yunfei;YU Weichi;ZHAO Jianing;WANG Shichao(School of Information Science and Engineering,Chongqing Jiaotong University,Chongqing 400074,China;Chest Hospital,Tianjin University,Tianjin 300072,China;College of Intelligence and Computing,Tianjin University,Tianjin 300072,China)

机构地区：[1]重庆交通大学信息科学与工程学院,重庆400074 [2]天津大学胸科医院,天津300072 [3]天津大学智能与计算学部,天津300072

出　　处：《无线电通信技术》2024年第5期831-842,共12页Radio Communications Technology

基　　金：国家自然科学基金(61602391);天津市科技计划项目(22JCZDJC00580)。

摘　　要：离线强化学习作为一种新兴范式,凭借其无需与环境交互即可利用大量离线数据进行策略学习的特性,展现出了很高的应用潜力和价值,特别是在医疗、自动驾驶等高风险领域中具有显著优势。从离线强化学习的基本概念、核心问题、主要方法依次展开,重点介绍多种缓解主要问题的方法:分布偏移的策略,包括约束目标策略与行为策略对齐、价值函数约束、模型不确定性量化以及基于模型的离线强化学习方法。讨论了目前离线强化学习的模拟环境以及重要应用场景。Offline reinforcement learning,as an emerging paradigm,leverages a vast amount of offline data for learning without the need of active interactions with the environment.It demonstrates high potential and value,especially in high-risk fields such as healthcare and autonomous driving.This review will sequentially unfold from the basic concepts of offline reinforcement learning,core issues,main methods,and focus on introducing various strategies to mitigate distributional shift.These include constraining target policy and behavior policy alignment,value function constraints,quantification of model uncertainty,and model-based offline reinforcement learning methods.Finally,the article discusses current simulation environments for offline reinforcement learning and significant application scenarios.

关 键 词：强化学习 离线强化学习 自动决策 外推误差 

分 类 号：TN919.23[电子电信—通信与信息系统]