基于深度强化学习TD3的PID参数自整定算法  

PID parameter self-tuning algorithm based on deep reinforcement learning TD3

作　　者：梁杰[1] 专祥涛[1,2] 严家政 LIANG Jie;ZHUAN Xiangtao;YAN Jiazheng(School of Electrical Engineering and Automation,Wuhan University,Wuhan 430072,China;Shenzhen Research Institute,Wuhan University,Shenzhen 518057,China)

机构地区：[1]武汉大学电气与自动化学院,湖北武汉430072 [2]武汉大学深圳研究院,广东深圳518057

出　　处：《武汉大学学报（工学版）》2024年第11期1635-1643,共9页Engineering Journal of Wuhan University

基　　金：深圳市知识创新计划项目(编号:JCYJ20170818144449801)。

摘　　要：传统的PID(proportional integral differential)算法在用于控制一些模型复杂、参数时变的对象时存在参数整定过程繁琐、控制性能不佳、无法解决控制对象实时变化状态的影响等问题。针对上述问题,提出了一种基于双延迟深度确定性策略梯度(twin delayed deep deterministic policy gradient,TDDDPG,以下简称TD3)算法的PID参数自整定算法。该算法将TD3算法与PID算法相结合,对TD3算法中的神经网络结构、奖励函数进行设计,能够实现控制器参数的自整定。以两轮直立车为实验对象,针对直立车的角度PID控制器进行参数整定实验。实验结果表明,与传统的参数整定算法(Z-N(Ziegler-Nichols)参数整定法)和基于强化学习的动态PID参数自整定算法相比,所提出的算法具有更优的控制效果,能够通过神经网络学习拟合更优的控制策略,提升控制器的动态响应性能和鲁棒性。When the traditional PID(proportional integral differential,PID)algorithm is used to control some objects with complex models and time-varying parameters,there are some problems,such as cumbersome parameter tuning process,poor control performance,and unable to solve the influence of the real-time changing state of the control objects.To solve these problems,a PID parameter self-tuning algorithm based on twin delayed deep deterministic policy gradient(TDDDPG,hereinafter referred to as TD3)algorithm is proposed.The algorithm combines TD3 algorithm with PID algorithm,and designs the neural network structure and reward function in TD3 algorithm,which can realize the self-tuning of controller parameters.Taking a twowheel upright vehicle as a experiment subject,the parameter tuning experiment of angle PID controller of upright vehicle is carried out.Experimental results show that compared with the traditional parameter tuning algorithm(Z-N parameter tuning method)and dynamic PID parameter self-tuning algorithm based on reinforcement learning,the proposed algorithm has better control effect,and can improve the dynamic response performance and robustness of the controller by learning and fitting better control strategies through neural networks.

关 键 词：深度强化学习 TD3算法 整定 控制器 直立车 

分 类 号：TP273.2[自动化与计算机技术—检测技术与自动化装置]