An Optimal Control-Based Distributed Reinforcement Learning Framework for A Class of Non-Convex Objective Functionals of the Multi-Agent Network  

作　　者：Zhe Chen Ning Li 

机构地区：[1]Department of Automation,Shanghai Jiao Tong University,Shanghai 200240 [2]Key Laboratory of System Control and Information Processing,Ministry of Education of China,Shanghai 200240 [3]Shanghai Engineering Research Center of Intelligent Control and Management,Shanghai 200240,China [4]Department of Automation,Tsinghua University,Beijing 100084,China [5]IEEE

出　　处：《IEEE/CAA Journal of Automatica Sinica》2023年第11期2081-2093,共13页自动化学报（英文版）

基　　金：supported in part by the National Natural Science Foundation of China(NSFC)(61773260);the Ministry of Science and Technology (2018YFB130590)。

摘　　要：This paper studies a novel distributed optimization problem that aims to minimize the sum of the non-convex objective functionals of the multi-agent network under privacy protection, which means that the local objective of each agent is unknown to others. The above problem involves complexity simultaneously in the time and space aspects. Yet existing works about distributed optimization mainly consider privacy protection in the space aspect where the decision variable is a vector with finite dimensions. In contrast, when the time aspect is considered in this paper, the decision variable is a continuous function concerning time. Hence, the minimization of the overall functional belongs to the calculus of variations. Traditional works usually aim to seek the optimal decision function. Due to privacy protection and non-convexity, the Euler-Lagrange equation of the proposed problem is a complicated partial differential equation.Hence, we seek the optimal decision derivative function rather than the decision function. This manner can be regarded as seeking the control input for an optimal control problem, for which we propose a centralized reinforcement learning(RL) framework. In the space aspect, we further present a distributed reinforcement learning framework to deal with the impact of privacy protection. Finally, rigorous theoretical analysis and simulation validate the effectiveness of our framework.

关 键 词：Distributed optimization MULTI-AGENT optimal control reinforcement learning(RL) 

分 类 号：TP18[自动化与计算机技术—控制理论与控制工程] TP309[自动化与计算机技术—控制科学与工程]