基于动态优先级的奖励优化模型  

Constrained Reward Optimization with Dynamic Preferences

作　　者：赵沛尧 黄蔚 ZHAO Peiyao;HUANG Wei(College of Computer Science and Technology, Soochow University, Suzhou 215006, China;Provincial Key Laboratory for Computer Information Processing Technology, Soochow University, Suzhou 215006, China;Soochow College, Soochow University, Suzhou 215006, China)

机构地区：[1]苏州大学计算机科学与技术学院,江苏苏州215006 [2]苏州大学江苏省计算机信息处理技术重点实验室,江苏苏州215006 [3]苏州大学东吴学院,江苏苏州215006

出　　处：《郑州大学学报（理学版）》2022年第1期62-68,共7页Journal of Zhengzhou University:Natural Science Edition

基　　金：国家自然科学基金项目(61303108);江苏省高校自然科学研究重大项目(17KJA520004);江苏省高校省级重点实验室(苏州大学)项目(KJS1524);苏州市应用基础研究计划工业部分(SYG201422)。

摘　　要：传统的约束马尔可夫决策过程(constrained Markov decision process,CMDP)模型用来解决多目标决策的困难,但该模型缺乏通用性并且只适用于离散环境。为了解决这一问题,提出一种基于CMDP的改进算法CRODP,与强化学习(reinforcement learning,RL)算法结合,将带约束问题建模为拉格朗日公式,其中包含着深度强化学习的策略参数。然后进一步推导出模型参数的更新公式,并将其作为深度强化学习神经网络的损失函数。模型在学习过程中动态调整约束与主要目标之间的权重以确保最终策略符合给定的约束。在若干机器人平台上与基准的深度强化学习算法进行比较,显示本算法得出的最优策略能够更好地满足相应约束。The traditional constrained Markov decision process(CMDP)model is used to solve the difficulties of multi-objective decision-making,but the model lacks universality and is only suitable for discrete environment.In order to solve this problem,an improved algorithm CRODP based on CMDP was proposed.Because of the combination of CRODP and reinforcement learning algorithm,CRODP could learn the general strategy.The algorithm modeled the constrained problem as a Lagrange formula,which contained the strategy parameters of deep reinforcement learning.Then,the updating formula of model parameters was further deduced as the loss function of the deep reinforcement learning neural network and the weight between constraints and main targets was dynamically adjusted in the learning process to ensure that the final strategy meets the given constraints.The results showed that the optimal strategy could meet the corresponding constraints better.

关 键 词：强化学习 深度学习 受限马尔可夫模型 动态优先级 机器人环境 

分 类 号：TP18[自动化与计算机技术—控制理论与控制工程]