强化学习在多阶段装备组合规划问题中的应用  

Application of reinforcement learning in multi-period weapon portfolio planning problems

作　　者：张骁雄 丁松 李明浩 丁鲲 王龙 义余江 ZHANG Xiaoxiong;DING Song;LI Minghao;DING Kun;WANG Long;YI Yujiang(The Sixty-third Research Institute, National University of Defense Technology, Nanjing 210007, China;School of Economics, Zhejiang University of Finance & Economics, Hangzhou 310018, China;College of Systems Engineering, National University of Defense Technology, Changsha 410073, China;Southwest Electronics and Telecommunication Technology Research Institute, Chengdu 610041, China)

机构地区：[1]国防科技大学第六十三研究所,江苏南京210007 [2]浙江财经大学经济学院,浙江杭州310018 [3]国防科技大学系统工程学院,湖南长沙410073 [4]西南电子电信技术研究所,四川成都610041

出　　处：《国防科技大学学报》2021年第5期127-136,共10页Journal of National University of Defense Technology

基　　金：国家自然科学基金资助项目(71901215,71901191);国防科技大学校科研计划资助项目(ZK20-46)。

摘　　要：针对多阶段武器装备组合规划中的选择难、规划难问题,提出基于多目标优化算法以及强化学习技术的混合优化方法。在各个阶段以装备组合效能最大和成本最小为准则,构建单阶段多目标优化模型,并设计基于非支配排序遗传算法的求解算法以生成各阶段的Pareto解,在此基础上建立多阶段的组合优化模型。通过强化学习的Q-Learning方法,在各阶段的Pareto解中采用探索或者利用两种模式,生成各阶段的装备组合,并指导下一阶段的装备选型,从而生成整个周期内的规划方案。通过对比实验分析,验证了所提模型和算法的有效性,能够为多阶段武器装备组合规划提供辅助决策。Aiming at the difficulties in the choosing and planning in multi-period weapon systems development problems,an optimization simulation approach combining multi-objective optimization algorithm and reinforcement learning technique was proposed.A multi-objective optimization model was built to maximize the capability and minimize the cost of weapon portfolios in each period.Moreover,a solving algorithm based on the non-dominated sorting genetic algorithm-Ⅲwas presented to obtain the Pareto set in each period,based on which an optimization model for multi-period problem was built.The Q-Learning method,one of the reinforcement learning algorithms,searches within the Pareto set using two different ways for the selection of weapon portfolios in each period,whose outcome is used for the selection in the next period and the optimization of the portfolios over the entire planning horizon.An illustrative example was studied to demonstrate the effectiveness of the proposed model and hybrid algorithm,which can support the decision making on the weapons development and planning.

关 键 词：武器装备 组合规划 非支配排序遗传算法 强化学习 Q-LEARNING 

分 类 号：O22[理学—运筹学与控制论] N94[理学—数学]