信息非完备下多航天器轨道博弈强化学习方法  

Reinforcement Learning Method for Multi-spacecraft Orbital Game with Incomplete Information

作　　者：王英杰 袁利 汤亮[1,3] 黄煌 耿远卓[1,3] WANG Yingjie;YUAN Li;TANG Liang;HUANG Huang;GENG Yuanzhuo(Beijing Institute of Control Engineering,Beijing 100094,China;China Academy of Space Technology,Beijing 100094,China;Science and Technology on Space Intelligent Control Laboratory,Beijing 100094,China)

机构地区：[1]北京控制工程研究所,北京100094 [2]中国空间技术研究院,北京100094 [3]空间智能控制技术重点实验室,北京100094

出　　处：《宇航学报》2023年第10期1522-1533,共12页Journal of Astronautics

基　　金：国家自然科学基金(U21B6001);国家自然科学基金青年基金(62203047);中国博士后科学基金(2022M722994)。

摘　　要：针对信息非完备约束下航天器轨道博弈难以自主决策的问题,基于多智能体强化学习提出一种多航天器轨道博弈决策方法。首先建立轨道博弈动力学和信息非完备约束。其次建立用于训练和决策的神经网络模型,依据分布式系统架构对网络的输入输出结构进行设计,并引入具有记忆功能的长短期记忆网络(LSTM),根据航天器轨道运动在时间、空间连续的属性,补偿位置、速度测量信息的非完备性。然后采用近端策略优化(PPO)算法开展红蓝左右互搏式学习训练。最后通过三组对比训练实验,验证了所提出的方法在信息非完备约束下能够有效增强学习训练过程的稳定性,并提升任务完成率和降低燃料消耗。Aiming at the problem that it’s difficult for spacecraft to make autonomous-decisions under the constraints of incomplete information,a multi-spacecraft orbital game decision-making method is proposed based on multi-agent reinforcement learning.Firstly,the dynamics of the orbital game and the constraints of incomplete information are modeled.Then neural networks are constructed for training and decision-making,the input and output of the networks are designed based on distributed system architecture,and long short-term memory(LSTM)networks with memory function are introduced,which compensate the incompleteness of position and velocity measurement information according to the continuity of orbit motion in space and time.Afterwards,self-play training mechanics using proximal policy optimization(PPO)is developed to train decision-making models.Finally,through three groups of comparative training experiments,it’s verified that the method proposed in this paper can enhance the stability of training effectively,improve mission completion rate and decrease fuel usage significantly.

关 键 词：航天器 信息非完备 轨道博弈 多智能体强化学习 长短期记忆网络 近端策略优化算法 

分 类 号：V249.328[航空宇航科学与技术—飞行器设计]