深度分层强化学习研究与发展  

Research and Development on Deep Hierarchical Reinforcement Learning

作　　者：黄志刚 刘全[1,2,3,4] 张立华 曹家庆 朱斐 HUANG Zhi-Gang;LIU Quan;ZHANG Li-Hua;CAO Jia-Qing;ZHU Fei(School of Computer Science and Technology,Soochow University,Suzhou 215006,China;Jiangsu Key Laboratory for Computer Information Processing Technology(Soochow University),Suzhou 215006,China;Key Laboratory of Symbolic Computation and Knowledge Engineering of Ministry of Education(Jilin University),Changchun 130012,China;Collaborative Innovation Center of Novel Software Technology and Industrialization(Nanjing),Nanjing 210093,China)

机构地区：[1]苏州大学计算机科学与技术学院,江苏苏州215006 [2]江苏省计算机信息处理技术重点实验室(苏州大学),江苏苏州215006 [3]符号计算与知识工程教育部重点实验室(吉林大学),吉林长春130012 [4]软件新技术与产业化协同创新中心(南京),江苏南京210093

出　　处：《软件学报》2023年第2期733-760,共28页Journal of Software

基　　金：国家自然科学基金(61772355,61702055,61876217,62176175);江苏省高等学校自然科学研究重大项目(18KJA520011,17KJA520004);吉林大学符号计算与知识工程教育部重点实验室资助项目(93K172014K04,93K172017K18,93K172021K08);苏州市应用基础研究计划工业部分(SYG201422);江苏高校优势学科建设工程资助项目。

摘　　要：深度分层强化学习是深度强化学习领域的一个重要研究方向,它重点关注经典深度强化学习难以解决的稀疏奖励、顺序决策和弱迁移能力等问题.其核心思想在于:根据分层思想构建具有多层结构的强化学习策略,运用时序抽象表达方法组合时间细粒度的下层动作,学习时间粗粒度的、有语义的上层动作,将复杂问题分解为数个简单问题进行求解.近年来,随着研究的深入,深度分层强化学习方法已经取得了实质性的突破,且被应用于视觉导航、自然语言处理、推荐系统和视频描述生成等生活领域.首先介绍了分层强化学习的理论基础;然后描述了深度分层强化学习的核心技术,包括分层抽象技术和常用实验环境;详细分析了基于技能的深度分层强化学习框架和基于子目标的深度分层强化学习框架,对比了各类算法的研究现状和发展趋势;接下来介绍了深度分层强化学习在多个现实生活领域中的应用;最后,对深度分层强化学习进行了展望和总结.Deep hierarchical reinforcement learning(DHRL)is an important research field in deep reinforcement learning(DRL).It focuses on sparse reward,sequential decision,and weak transfer ability problems,which are difficult to be solved by classic DRL.DHRL decomposes complex problems and constructs a multi-layered structure for DRL strategies based on hierarchical thinking.By using temporal abstraction,DHRL combines lower-level actions to learn semantic higher-level actions.In recent years,with the development of research,DHRL has been able to make breakthroughs in many domains and shows a strong performance.It has been applied to visual navigation,natural language processing,recommendation system and video description generation fields in real world.In this study,the theoretical basis of hierarchical reinforcement learning(HRL)is firstly introduced.Secondly,the key technologies of DHRL are described,including hierarchical abstraction techniques and common experimental environments.Thirdly,taking the option-based deep hierarchical reinforcement learning framework(O-DHRL)and the subgoal-based deep hierarchical reinforcement learning framework(G-DHRL)as the main research objects,those research status and development trend of various algorithms are analyzed and compared in detail.In addition,a number of DHRL applications in real world are discussed.Finally,DHRL is prospected and summarized.

关 键 词：人工智能 强化学习 深度强化学习 半马尔可夫决策过程 深度分层强化学习 

分 类 号：TP18[自动化与计算机技术—控制理论与控制工程]