Sampled-data control through model-free reinforcement learning with effective experience replay  

作　　者：Bo Xiao H.K.Lam Xiaojie Su Ziwei Wang Frank P.-W.Lo Shihong Chen Eric Yeatman 

机构地区：[1]Department of Electrical and Electronic Engineering,Imperial College London,London SW72AZ,UK [2]Department of Engineering,King’s College London,London WC2B 4BG,UK [3]Department of Automation,Chongqing University,Shapingba District,Chong Qing,China [4]School of Engineering,Lancaster University,LA14YW,UK [5]Hamlyn Centre,Imperial College London,London SW72AZ,UK

出　　处：《Journal of Automation and Intelligence》2023年第1期20-30,共11页自动化与人工智能（英文）

基　　金：supported by Imperial College London,UK,King’s College London,UK and Engineering and Physical Sciences Research Council(EPSRC),UK.

摘　　要：Reinforcement Learning(RL)based control algorithms can learn the control strategies for nonlinear and uncertain environment during interacting with it.Guided by the rewards generated by environment,a RL agent can learn the control strategy directly in a model-free way instead of investigating the dynamic model of the environment.In the paper,we propose the sampled-data RL control strategy to reduce the computational demand.In the sampled-data control strategy,the whole control system is of a hybrid structure,in which the plant is of continuous structure while the controller(RL agent)adopts a discrete structure.Given that the continuous states of the plant will be the input of the agent,the state–action value function is approximated by the fully connected feed-forward neural networks(FCFFNN).Instead of learning the controller at every step during the interaction with the environment,the learning and acting stages are decoupled to learn the control strategy more effectively through experience replay.In the acting stage,the most effective experience obtained during the interaction with the environment will be stored and during the learning stage,the stored experience will be replayed to customized times,which helps enhance the experience replay process.The effectiveness of proposed approach will be verified by simulation examples.

关 键 词：Reinforcement learning Neural networks Sampled-data control MODEL-FREE Effective experience replay 

分 类 号：TP181[自动化与计算机技术—控制理论与控制工程]