基于强化学习的离散事件系统最优定向监控  

Optimal Directed Control of Discrete Event Systems Based on Reinforcement Learning

作　　者：胡瑜洪 王德光 杨明 王玺 HU Yu-hong;WANG De-guang;YANG Ming;WANG Xi(The Electrical Engineering College,Guizhou University,Guiyang,Guizhou 550025,China;School of Eletro-Mechanical Engineering,Xidian University,Xi'an,Shaanxi 710071,China)

机构地区：[1]贵州大学电气工程学院,贵州贵阳550025 [2]西安电子科技大学机电工程学院,陕西西安710071

出　　处：《电子学报》2024年第9期3172-3184,共13页Acta Electronica Sinica

基　　金：国家自然科学基金(No.52265066,No.62203132);贵州省省级科技计划资助项目(No.黔科合基础-ZK[2022]一般103);贵州省教育厅青年科技人才成长项目(No.黔教合KY字[2022]138号);贵州省教育厅创新群体(No.黔科合支撑[2021]012);贵州大学科研基金资助项目(No.贵大特岗合字[2021]04号)。

摘　　要：对于多个可控事件(控制指令)允许同时执行的情形,离散事件系统的监控器进行随机选择.然而,在实际应用中,如交通调度、机器人路径规划,可控事件的定向选择和数值优化是必须要考虑和解决的两个问题.对此,引入一种优化机制量化控制成本,将监督控制理论与强化学习结合,提出一种基于强化学习的离散事件系统最优定向监控器求解方法,使被控系统实现以下三个目标:(1)遵循安全性和活性控制规范;(2)每个状态下至多允许一个可控事件执行;(3)从初始状态到标记状态事件执行累计成本最小.首先,建立系统和控制规范的自动机模型,做同步积运算后可得到目标模型,通过定义的成本函数为目标模型中每个事件的执行赋予成本.其次,利用监督控制理论求解无阻塞且行为最大许可的监控器.最后,将监控器转化为马尔可夫决策过程,并利用Q学习算法求解出最优定向监控器.使用单向列车导轨控制案例和多轨道列车控制案例验证所提方法的有效性和正确性.仿真结果表明,所提出方法能够实现系统的无阻塞定向控制,并且使得定向监控器的数值成本最小.In the case that several controllable events(control commands)are allowed to execute simultaneously,the supervisor in the framework of discrete event systems(DESs)selects one randomly.However,in practical applications,such as traffic scheduling and robot path planning,the problems of directed control and numerical optimization should be considered.This paper introduces an optimization mechanism to quantify the control cost and combines supervisory control theory(SCT)with reinforcement learning.A systematic procedure is proposed to synthesize the optimal directed supervisor of a DES based on reinforcement learning,which makes the controlled system achieve the following three goals:(1)the con⁃trol specifications relevant to security and liveness are not violated;(2)at most one controllable event can be executed at each state;(3)the cumulative cost of event execution from the initial state to a mark state is minimal.First,given the autom⁃aton models of the plant and specifications,the target automaton model is obtained by the synchronous operation of these two models;a cost function is defined and assigns the execution cost for each event in the target model.Second,the nonblocking and maximally permissive supervisor is synthesized by SCT.Finally,the supervisor is transformed into a Markov decision process and then the Q-learning algorithm is utilized to compute the optimal directed supervisor.Two applications are used to verify the effectiveness and correctness of the proposed method.The simulation results show that the proposed method can realize the directed control of the system,and the numerical cost of the directed supervisor is minimized.

关 键 词：离散事件系统 定向监控器 强化学习 最优控制 数值优化 交通系统 

分 类 号：TP301[自动化与计算机技术—计算机系统结构]