Supervisory control of the hybrid off-highway vehicle for fuel economy improvement using predictive double Q-learning with backup models  

基于备份预测模型和双Q学习算法的混动非公路车辆电量保持控制策略

作　　者：SHUAI Bin LI Yan-fei ZHOU Quan XU Hong-ming SHUAI Shi-jin 帅斌;李雁飞;周泉;徐宏明;帅石金(Department of Mechanical Engineering,the University of Birmingham,B152TT,UK;State Key Laboratory of Automotive Energy and Safety,Tsinghua University,Beijing 100083,China)

机构地区：[1]Department of Mechanical Engineering,the University of Birmingham,B152TT,UK [2]State Key Laboratory of Automotive Energy and Safety,Tsinghua University,Beijing 100083,China

出　　处：《Journal of Central South University》2022年第7期2266-2278,共13页中南大学学报（英文版）

基　　金：Project(KF2029)supported by the State Key Laboratory of Automotive Safety and Energy(Tsinghua University),China;Project(102253)supported partially by the Innovate UK。

摘　　要：This paper studied a supervisory control system for a hybrid off-highway electric vehicle under the chargesustaining(CS)condition.A new predictive double Q-learning with backup models(PDQL)scheme is proposed to optimize the engine fuel in real-world driving and improve energy efficiency with a faster and more robust learning process.Unlike the existing“model-free”methods,which solely follow on-policy and off-policy to update knowledge bases(Q-tables),the PDQL is developed with the capability to merge both on-policy and off-policy learning by introducing a backup model(Q-table).Experimental evaluations are conducted based on software-in-the-loop(SiL)and hardware-in-the-loop(HiL)test platforms based on real-time modelling of the studied vehicle.Compared to the standard double Q-learning(SDQL),the PDQL only needs half of the learning iterations to achieve better energy efficiency than the SDQL at the end learning process.In the SiL under 35 rounds of learning,the results show that the PDQL can improve the vehicle energy efficiency by 1.75%higher than SDQL.By implementing the PDQL in HiL under four predefined real-world conditions,the PDQL can robustly save more than 5.03%energy than the SDQL scheme.面向非公路混动车辆“无模型”电量保持控制,提出了一种融合备份预测模型和双Q学习算法的学习系统(PDQL),通过更快、更稳健的机器学习,不断优化车辆的能量效率。与现有的标准无模型控制策略的单独进行同策略学习方法或者单独进行异策略学习方法对智能体经验库进行更新不同,本文提出的PDQL将其中一个Q表看做一个备份经验库,以融合同策略学习过程与异策略学习过程。基于所研究车辆的实时模型,利用软件在环和硬件在环测试平台,对PDQL系统进行实验评估。通过对比标准双Q学习(SDQL),PDQL只需要一半的时间就能达到SDQL所能获得的最佳能量效率。经过35轮离线学习,PDQL可以在SDQL的基础上,再提高1.75%的车辆能量效率。对PDQL进行鲁棒性测试,在模拟真实驾驶条件下,与基于SDQL算法的车辆相比,可以节省5.03%以上的能量。

关 键 词：supervisory charge-sustaining control hybrid electric vehicle reinforcement learning predictive double Q-learning 

分 类 号：TP18[自动化与计算机技术—控制理论与控制工程] U469.7[自动化与计算机技术—控制科学与工程]