基于解耦型MADDPG的无人机集群自主跟踪与避障  

Autonomous tracking and obstacle avoidance of UAV swarms based on decomposed MADDPG

作　　者：文超 董文瀚[1] 解武杰[1] 蔡鸣 胡多修 WEN Chao;DONG Wenhan;XIE Wujie;CAI Ming;HU Duoxiu(Air Force Engineering University,Xi'an 710038,China)

机构地区：[1]空军工程大学,陕西西安710038

出　　处：《飞行力学》2022年第6期24-31,共8页Flight Dynamics

摘　　要：考虑单无人机在复杂空域环境下对地面动目标执行跟踪任务的局限性,采用多智能体深度强化学习(MADRL)方法对无人机集群目标自主跟踪问题进行了研究。首先,基于随机博弈过程设计联合状态空间、动作空间和奖惩机制,并由此建立了无人机集群三维自主机动模型。其次,考虑MADRL的稀疏回报问题,设计了引导型奖励函数,增强了算法收敛性能。接着,为提高集群学习效率,设计了相应的解耦型奖励函数和神经网络结构,并采用解耦型多智能体深度确定性策略梯度(MADDPG)算法对模型进行自适应训练,以生成无人机集群自主跟踪与避障最优机动策略。最后,开展了仿真验证。结果表明:基于MADRL方法的无人机集群能更好地满足复杂空域环境下目标跟踪任务的需求;相比于MADDPG,解耦型MADDPG算法具有更强的准确性和实时性。Considering the limitations of single UAV in performing tracking missions on ground moving targets in complex airspace environments, the multi-agent deep reinforcement learning(MADRL) method was used to study the problem of UAV swarms tracking targets autonomously. Firstly, the joint state space, action space, and reward and punishment mechanism were designed based on the stochastic game process, and 3 D autonomous maneuvering model of UAV swarms was established. Secondly, considering the sparse return problem of MADRL, the guided reward function was designed to enhance the algorithm convergence performance. Then, the decomposed reward functions and neural network structure were developed to improve the learning efficiency of swarms, and the decomposed MADDPG was used to train the model adaptively so as to generate the optimal maneuvering strategy for autonomous tracking and obstacle avoidance of UAV swarms. Finally, the simulation validation was conducted. The results show that MADRL-based UAV swarms can better meet the requirements of tracking missions in complex airspace environments. Compared with MADDPG, the decomposed MADDPG has stronger accuracy and real-time performance.

关 键 词：无人机集群 多智能体深度强化学习 自主跟踪 避障 随机博弈过程 

分 类 号：V279[航空宇航科学与技术—飞行器设计] V249