基于混合动作的空战分层强化学习决策算法  

Hierarchical decision algorithm for air combat with hybrid action based on deep reinforcement learning

作　　者：李佐龙 朱纪洪 匡敏驰 张杰[2] 任洁 LI Zuolong;ZHU Jihong;KUANG Minchi;ZHANG Jie;REN Jie(Department of Precision Instrument,Tsinghua University,Beijing 100084,China;AVIC Chengdu Flight Design and Research Institute,Chengdu 610091,China)

机构地区：[1]清华大学精密仪器系,北京100084 [2]航空工业成都飞机设计研究所,成都610091

出　　处：《航空学报》2024年第17期156-173,共18页Acta Aeronautica et Astronautica Sinica

摘　　要：智能空战是世界主要军事强国的研究热点。为解决超视距空战博弈机动决策问题,提出了基于深度强化学习的超视距空战分层决策算法。在该决策算法中,使用适合于超视距空战的机动动作集,对飞机的航迹和姿态进行控制。为了扩大模型的动作空间,提升模型的决策能力,将空战的动作空间进行分层,建模为多维离散的动作空间。针对空战中稀疏奖励的问题,设计了一套综合考虑位置优势、武器发射和武器威胁等要素的奖励函数,用于引导智能体向最优策略收敛。搭建了完整的数字孪生空战仿真环境和空战专家系统,在仿真环境中训练决策算法,并通过与专家系统的对抗,对决策算法进行评估。实验结果表明:决策算法具备超视距空战自主决策的能力,能够根据战场态势,进行灵活的机动决策,在与专家系统对抗的过程中具有一定的优势。Intelligent air combat is a hot research topic among countries with strong military power in the world.To solve the maneuver decision problem of air combat Beyond Visual Range(BVR),we propose the hierarchical deci⁃sion algorithm based on deep reinforcement learning.In the decision algorithm,we use the maneuver set appropriate to the BVR air combat to control the trajectory and the attitude of the aircraft.To expand the action space of the model and increase its decision-making ability,we hierarchize the action space and model it as the multi-discrete one.To solve the problem of sparse reward in air combat,we design a set of reward function taking into consideration the fac⁃tors including the position advantage,weapon launching,and weapon threat,which can guide the agent to converge to the optimal policy.We also build a complete digital-twin simulation environment for air combat and an expert sys⁃tem.The decision algorithm is trained in the simulation environment,and is evaluated by fighting with the expert sys⁃tem.The experiment results indicate that the decision algorithm proposed has the ability to make autonomous and flex⁃ible decisions in BVR air combat based on current situations,and has some advantages against the expert system.

关 键 词：超视距空战 智能决策 深度强化学习 近端策略优化 机动动作 分层决策 

分 类 号：V249.4[航空宇航科学与技术—飞行器设计]