基于深度强化学习的平流层浮空器高度控制  

Altitude control of stratospheric aerostat based on deep reinforcement learning

作　　者：张经伦 杨希祥[1] 邓小龙 郭正[1] 翟嘉琪 ZHANG Jinglun;YANG Xixiang;DENG Xiaolong;GUO Zheng;ZHAI Jiaqi(College of Aerospace Science and Engineering,National University of Defense Technology,Changsha 410073,China)

机构地区：[1]国防科技大学空天科学学院,长沙410073

出　　处：《北京航空航天大学学报》2023年第8期2062-2070,共9页Journal of Beijing University of Aeronautics and Astronautics

基　　金：国家自然科学基金(52272445);湖南省自然科学基金(2023JJ100056)。

摘　　要：为研究基于深度强化学习的平流层浮空器高度控制问题。建立平流层浮空器动力学模型,提出一种基于深度Q网络(DQN)算法的平流层浮空器高度控制方法,以平流层浮空器当前速度、位置、高度差作为智能体的观察状态,副气囊鼓风机开合时间作为智能体的输出动作,平流层浮空器非线性动力学模型与扰动风场作为智能体的学习环境。所提方法将平流层浮空器的高度控制问题转换为未知转移概率下连续状态、连续动作的强化学习过程,兼顾随机风场扰动与速度变化约束,实现稳定的变高度控制。仿真结果表明:考虑风场环境对浮空器影响下,DQN算法控制器可以很好的实现变高度的跟踪控制,最大稳态误差约为10 m,与传统比例积分微分(PID)控制器对比,其控制效果和鲁棒性更优。A dynamic model of the stratospheric aerostat was built with the goal of controlling the aerostat's altitude while taking air temperature into consideration,and a method based on the deep Q-network(DQN)algorithm was developed.Due to the difficulty in predicting the stratospheric wind field and the physical model of the aerostat itself being unknown,most model-based control methods cannot solve the problem of long-term altitude control of the stratospheric aerostat.For this reason,the altitude control problem of the stratospheric aerostat is transformed into a continuous state and continuous action reinforcement learning process with unknown transition probability.The DQN algorithm combined with reinforcement learning and neural network can solve such problems well.The simulation results show that considering the influence of the wind field environment on the aerostat,the DQN algorithm controller can well realize the tracking control of variable altitude,and the maximum error is about 10 m.Compared with the traditional proportional inteyral derivative(PID)controller,the deep reinforcement learning algorithm proposed in this paper has a better control effect and robustness.

关 键 词：临近空间 平流层浮空器 高度控制 深度强化学习 PID控制器 

分 类 号：V472[航空宇航科学与技术—飞行器设计] TB553[理学—物理]