近端策略优化的城市环境多智能体协作对抗方法  

Multi-agent cooperative confrontation with proximal policy optimization in urban environments

作　　者：米广铭 张辉[1,2] 张菁[1,2] 卓力[1,2] MI Guangming;ZHANG Hui;ZHANG Jing;ZHUO Li(School of Information Science and Technology,Beijing University of Technology,Beijing 100124,China;Beijing Key Laboratory of Computational Intelligence and Intelligent System,Beijing University of Technology,Beijing 100124,China)

机构地区：[1]北京工业大学信息科学技术学院,北京100124 [2]北京工业大学计算智能与智能系统北京市重点实验室,北京100124

出　　处：《通信学报》2025年第3期94-108,共15页Journal on Communications

基　　金：北京市自然科学基金资助项目(No.L247025)。

摘　　要：城市环境由于其地理空间的复杂性及动态变化性,往往会令指挥系统变得低效且短视。针对该问题,提出了一种近端策略优化城市环境的多智能体协作对抗方法。首先,在建立完善的城市对抗环境的基础上,使用近端策略优化的演员-评论员网络算法进行求解;其次,针对多对一的评论网络采用嵌入方法来解决空间维度不同的异构智能体决策评价问题;再次,在近端策略优化的基础上,增加了自适应采样来辅助策略的更新;最后,对演员网络进行权重继承操作以帮助智能体迅速接管相应的任务。实验结果表明,相较于其他方法,所提方法的奖励回报提高了22.67%,收敛速度加快了8.14%,不仅可以满足城市环境下多个智能体协作对抗的决策,还能够兼容多异构智能体的协作对抗。To address the issue that urban environments often make command systems inefficient and inflexible due to their geospatial complexity and dynamic changes,a multi-agent cooperative confrontation method with proximal policy optimization for urban environments was proposed.First,on the basis of establishing a comprehensive urban confrontation environment,the AC(actor-critic)network with proximal policy optimization was used to solve the problem.Then,aiming at the multi-to-one critic network,an embedding method was adopted to address the issue of evaluating the decision-making of heterogeneous agents with different spatial dimensions.Furthermore,adaptive sampling was added to assist in the updating of proximal policy optimization.Finally,the weights of the actor network were inherited to help agents quickly take over the corresponding tasks.Experimental results show that the proposed method improves 22.67%reward and 8.14%convergence rate compared to other methods,which not only meets the decision-making of multiple agents’cooperative confrontation in urban environments,but also is compatible with the cooperative confrontation of multiple heterogeneous agents.

关 键 词：深度强化学习 多智能体 协作对抗 近端策略优化 城市环境 

分 类 号：TP3[自动化与计算机技术—计算机科学与技术]