基于深度Q网络的平衡杆优化控制实验教学案例设计  

Design of an Experimental Teaching Case for Balance Pole Optimization Control Based on Deep Q-Network

作　　者：钟珊[1,2] 杜鑫 司亚利 刘井莲 严卫 ZHONG Shan;DU Xin;SI Yali;LIU Jinglian;YAN Wei(School of Computer Science and Engineering,Changshu Institute of Technology,Changshu 215500,Jiangsu,China;State Key Laboratory for Novel Software Technology,Nanjing University,Nanjing 210023,China;School of Electronics and Information Engineering,Suzhou University of Science and Technology,Suzhou 215009,Jiangsu,China)

机构地区：[1]常熟理工学院计算机科学与工程学院,江苏常熟215500 [2]南京大学计算机软件新技术全国重点实验室,南京210023 [3]苏州科技大学电子信息与工程学院,江苏苏州215009

出　　处：《实验室研究与探索》2025年第1期146-152,190,共8页Research and Exploration In Laboratory

基　　金：国家自然科学基金面上项目(62376041);国家自然科学基金青年项目(621023471);中国高等教育学会“十四五”规划专项课题(21JSYB16);江苏省高等教育质量保障与评价研究课题(2021-C01);南京大学计算机软件新技术全国重点实验室开放课题项目(KFKT2024B51)。

摘　　要：为了使学生更深入地掌握采用强化学习求解任务最优策略的方法,基于深度Q网络设计了平衡杆的控制优化实验。将平衡杆实验环境建模为马尔可夫决策模型,从深度Q网络结构、探索策略、奖励函数、经验回放池更新和目标Q网络更新等角度进行了实验方案的设计。实验评估综合考虑了算法有效性、Q值估计的准确性、样本效率和Q网络的损失估计。在此基础上,将其与基准算法Q学习算法在累计奖赏和学习效率上进行了对比。该实验不仅可以帮助学生学会综合应用强化学习知识,也能帮助其掌握采用深度Q网络算法来求解离散动作空间任务的最优策略。In order to enable students to gain a deeper understanding of applying reinforcement learning to solve optimal strategies for tasks,an optimization experiment for balance pole was designed based on deep Q-network(DQN).The aim is to learn how to apply actions to a cart so that the pole keeps balancing.Firstly,the balance pole experimental environment was modeled as a Markov Decision Process(MDP).Then,the experimental design was conducted from the perspectives of DQN structure,exploration strategy,reward function,experience replay buffer update,and target Q-network update.The evaluation of the experiment comprehensively considered the effectiveness of the algorithm,the accuracy of Q-value estimation,sample efficiency,and the loss estimation of the Q-network.On this basis,it was compared with the benchmark algorithm,Q-learning,in terms of cumulative reward and learning efficiency.This experiment not only helps students comprehensively apply reinforcement learning knowledge and better understand to use the DQN algorithm to solve optimal strategies for tasks with discrete action spaces.

关 键 词：强化学习 深度Q网络 Q学习 平衡杆 实验设计 

分 类 号：TP391.4[自动化与计算机技术—计算机应用技术]