一种优先级扫描的Dyna结构优化算法  

An Optimized Dyna Architecture Algorithm with Prioritized Sweeping

作　　者：孙洪坤[1] 刘全[1,2] 傅启明[1] 肖飞[1] 高龙[1] 

机构地区：[1]苏州大学计算机科学与技术学院,江苏苏州215006 [2]符号计算与知识工程教育部重点实验室(吉林大学),长春130012

出　　处：《计算机研究与发展》2013年第10期2176-2184,共9页Journal of Computer Research and Development

基　　金：国家自然科学基金项目(61070223;61103045;61272005;61272244);江苏省自然科学基金项目(BK2012616);江苏省高校自然科学研究项目(09KJA520002;09KJB520012);吉林大学符号计算与知识工程教育部重点实验室基金项目(93K172012K04)

摘　　要：不确定环境的时序决策问题是强化学习研究的主要内容之一,agent的目标是最大化其与环境交互过程中获得的累计奖赏值.直接学习方法寻找最优策略的算法收敛效率较差,而采用Dyna结构将学习与规划并行集成,可提高算法的收敛效率.为了进一步提高传统Dyna结构的收敛速度和收敛精度,提出了Dyna-PS算法,并在理论上证明了其收敛性.该算法在Dyna结构规划部分使用优先级扫描算法的思想,对优先级函数值高的状态优先更新,剔除了传统值迭代、策略迭代过程中不相关和无更新意义的状态更新,提升了规划的收敛效率,从而进一步提升了Dyna结构算法的性能.将此算法应用于一系列经典规划问题,实验结果表明,Dyna-PS算法有更快的收敛速度和更高的收敛精度,且对于状态空间的增长具有较强的鲁棒性.Reinforcement learning involves sequential decision making in model-flee environments. The aim of the agent is to maximize the accumulated reward of acting in its environment over an extended period of time. Finding the optimal policy in direct RL may be very slow. To speed up converging, one often-used solution is the integration of learning with planning. In order to further improve the convergence time and convergence precision of the Dyna structure algorithm, an optimized Dyna structure algorithm with prioritized sweeping named Dyna-PS is proposed, and its proof of convergence in theory is given. The key idea of Dyna PS is integrating prioritized sweeping method in Dyna architecture so as to update the states according to their priority functions in the planning part. Moreover, it omits the insignificant and unrelated states＇ updating which are often updated in traditional value iteration and policy iteration. Achieved experiment results show that the Dyna-PS algorithm has better convergence performance and robustness for state space growth when it is applied to the maze experiment scenario and a series of classical AI programming problems.

关 键 词：强化学习 时序决策 优先级扫描 Dyna结构 Dyna—PS 

分 类 号：TP181[自动化与计算机技术—控制理论与控制工程]