一种基于投票法的多策略多目标强化学习算法  

A Multi-Policy Multi-Objective Reinforcement Learning Algorithm Based on Voting Method

作　　者：王立佳 朱正伟[1] 诸燕平[1] 朱晨阳[1] WANG Li-jia;ZHU Zheng-wei;ZHU Yan-ping;ZHU Chen-yang(Aliyun School of Big Data,Changzhou University,Changzhou Jiangsu 213000,China)

机构地区：[1]常州大学阿里云大数据学院,江苏常州213000

出　　处：《计算机仿真》2023年第4期341-345,406,共6页Computer Simulation

基　　金：国家自然科学基金(61801055);2021年常州市第十三批科技计划(应用基础研究)(CJ20210123)。

摘　　要：针对多目标强化学习中的多策略算法收敛不稳定的问题,提出了一种基于Sarsa算法框架的多目标帕累托Sarsa算法(Multi-Pareto Sarsa, MPS)。MPS算法以向量集的形式更新行为价值,使用一种新的基于投票法的集合评估机制代替传统的行为策略,智能体通过基于投票法的集合评估机制评估行为空间中每个行为对应的行为价值向量集,然后根据评估结果选择当前状态下的最优行为。实验结果表明,MPS算法与已有的MPQ、PQL算法相比有较好的收敛性,且MPS算法与MPQ算法相比在超体积性能上有很大的提升。To solve the problem of unstable convergence of multi-policy algorithms in multi-objective reinforcement learning,this paper proposes a multi-objective Sarsa algorithm based on the standard Sarsa algorithm framework(Multi-Pareto Sarsa,MPS).The MPS algorithm updates the behavior value in the form of vector sets,and uses a new voting-based set evaluation mechanism to replace the traditional behavior strategy.The agents use a votingbased set evaluation mechanism to evaluate the behavior value vector set corresponding to each action in the action space.Then the agents select the best behavior in the current state according to the evaluation result.The experimental results show that the MPS algorithm has better convergence than the existing MPQ and PQL algorithms,and the MPS algorithm has a great improvement in hypervolume performance compared with the MPQ algorithm.

关 键 词：多目标 强化学习 多策略 投票法 集合评估机制 

分 类 号：TP181[自动化与计算机技术—控制理论与控制工程]