基于批量递归最小二乘的自然Actor-Critic算法  

Natural Actor-Critic based on batch recursive least-squares

作　　者：王国芳[1] 方舟[1] 李平[1] 

机构地区：[1]浙江大学航空航天学院,浙江杭州310027

出　　处：《浙江大学学报（工学版）》2015年第7期1335-1342,共8页Journal of Zhejiang University：Engineering Science

基　　金：国家自然科学基金资助项目(61004066);浙江省自然科学基金资助项目(LY15F030005)

摘　　要：为了减轻Actor-Critic结构中智能体用最小二乘法估计自然梯度时的在线运算负担,提高运算实时性,提出新的学习算法:NAC-BRLS.该算法在Critic中利用批量递归最小二乘法估计自然梯度,根据估计得到的梯度乐观地更新策略.批量递归最小二乘法的引入使得智能体能根据自身运算能力自由调整各批次运算的数据量,即每次策略估计时使用的数据量,在全乐观和部分乐观之间进行权衡,大大提高了NAC-LSTD算法的灵活性.山地车仿真实验表明,与NAC-LSTD算法相比,NAC-BRLS算法在保证一定收敛性能的前提下,能够明显降低智能体的单步平均运算负担.The algorithm called natural actor-critic based on batch recursive least-squares（NAC-BRLS）was proposed in order to reduce the online computation burden of the agent and improve the real-time operation.The algorithm employed batch recursive least-squares in Critic to evaluate the natural gradient,and performed optimistic update in Actor by the estimated natural gradient.The use of batch recursive leastsquares enables the agent to adjust the date size of every batch according to its operational capability.A trade-off between fully optimistic and partially optimistic was made,improving the flexibility of NACLSTD.Simulation results in mountain car show that NAC-BRLS largely reduces the computational complexity without obviously affecting the convergence property compared with NAC-LSTD.

关 键 词：自然梯度 Actor-Critic 批次更新 递归最小二乘 

分 类 号：TP18[自动化与计算机技术—控制理论与控制工程]