Joint UAV trajectory and communication design with heterogeneous multi-agent reinforcement learning  

作　　者：Xuanhan ZHOU Jun XIONG Haitao ZHAO Xiaoran LIU Baoquan REN Xiaochen ZHANG Jibo WEI Hao YIN 

机构地区：[1]College of Electronic Science and Technology,National University of Defense Technology,Changsha 410073,China [2]Systems Engineering Institute,Academy of Military Sciences PLA,Beijing 100091,China

出　　处：《Science China(Information Sciences)》2024年第3期221-241,共21页中国科学（信息科学）（英文版）

基　　金：supported in part by National Natural Science Foundation of China (Grant Nos.62371462,61931020,62101569,U19B2024);Natural Science Foundation of Hunan Province (Grant No.2022JJ10068);Science and Technology Innovation Program of Hunan Province (Grant No.2022RC1093)。

摘　　要：Unmanned aerial vehicles(UAVs)are recognized as effective means for delivering emergency communication services when terrestrial infrastructures are unavailable.This paper investigates a multiUAV-assisted communication system,where we jointly optimize UAVs'trajectories,user association,and ground users(GUs)'transmit power to maximize a defined fairness-weighted throughput metric.Owing to the dynamic nature of UAVs,this problem has to be solved in real time.However,the problem's non-convex and combinatorial attributes pose challenges for conventional optimization-based algorithms,particularly in scenarios without central controllers.To address this issue,we propose a multi-agent deep reinforcement learning(MADRL)approach to provide distributed and online solutions.In contrast to previous MADRLbased methods considering only UAV agents,we model UAVs and GUs as heterogeneous agents sharing a common objective.Specifically,UAVs are tasked with optimizing their trajectories,while GUs are responsible for selecting a UAV for association and determining a transmit power level.To learn policies for these heterogeneous agents,we design a heterogeneous coordinated QMIX(HC-QMIX)algorithm to train local Q-networks in a centralized manner.With these well-trained local Q-networks,UAVs and GUs can make individual decisions based on their local observations.Extensive simulation results demonstrate that the proposed algorithm outperforms state-of-the-art benchmarks in terms of total throughput and system fairness.

关 键 词：unmanned aerial vehicle(UAV) trajectory design resource allocation multi-agent deep reinforcement learning(MADRL) heterogeneous agents 

分 类 号：TN929.5[电子电信—通信与信息系统] TP18[电子电信—信息与通信工程] V279[自动化与计算机技术—控制理论与控制工程]