移动边缘计算中基于Actor-Critic深度强化学习的任务调度方法  

Task scheduling method with Actor-Critic deep reinforcement learning in mobile edge computing

作　　者：黄一帆 曾旺 陈哲毅 于正欣 苗旺 HUANG Yifan;ZENG Wang;CHEN Zheyi;YU Zhengxin;MIAO Wang(College of Computer and Data Science,Fuzhou University,Fuzhou Fujian 350116,China;Key Laboratory of Spatial Data Mining&Information Sharing,Ministry of Education(Fuzhou University),Fuzhou Fujian 350002,China;Fujian Key Laboratory of Network Computing and Intelligent Information Processing(Fuzhou University),Fuzhou Fujian 350116,China;School of Computing and Communications,Lancaster University,Lancaster LA14YW,UK;School of Engineering,Computing and Mathematics,University of Plymouth,Plymouth,PL48AA,UK)

机构地区：[1]福州大学计算机与大数据学院,福州350116 [2]空间数据挖掘与信息共享教育部重点实验室(福州大学),福州350002 [3]福建省网络计算与智能信息处理重点实验室(福州大学),福州350116 [4]兰卡斯特大学计算与通信学院,英国兰卡斯特LA14YW [5]普利茅斯大学工程、计算与数学学院,英国普利茅斯PL48AA

出　　处：《计算机应用》2024年第S01期150-155,共6页journal of Computer Applications

基　　金：国家自然科学基金资助项目(62202103);中央引导地方科技发展资金资助项目(2022L3004);福建省财政厅科研专项(83021094);福建省科技经济融合服务平台项目(2023XRH001);福厦泉国家自主创新示范区协同创新平台项目(2022FX5)。

摘　　要：移动边缘计算(MEC)通过将计算与存储资源部署至网络边缘,有效降低了任务响应时间并提高了资源利用率。由于MEC系统状态的动态性和用户需求的多变性,如何进行有效的任务调度面临着巨大的挑战,不合理的任务调度策略将严重影响系统的整体性能。现有工作通常对任务采用平均分配资源或基于规则的策略,不能有效地处理动态的MEC环境,这可能造成过多的资源消耗,进而导致服务质量(QoS)下降。针对上述重要问题,提出了一种MEC中基于Actor-Critic深度强化学习的任务调度方法(TSAC)。首先,提出了一种面向边缘环境的任务调度模型并将任务等待时间和任务完成率作为优化目标;其次,基于所提系统模型与深度强化学习框架,将联合优化问题形式化为马尔可夫决策过程;最后,基于近端策略优化方法,设计了一种新型的掩码机制,在避免智能体做出违反系统约束的动作和策略突变的同时提高了TSAC的收敛性能。基于谷歌集群真实运行数据集进行仿真实验,与深度Q网络方法相比,至少降低6%的任务等待时间,同时提高4%的任务完成率,验证了的可行性和有效性。Mobile Edge Computing(MEC)can effectively reduce task response time and improve resource utilization by deploying computing and storage resources at the network edge.The dynamic state of MEC system and the variability of user demands make the task scheduling become extremely challenging,and unreasonable scheduling strategies would seriously affect the overall system performance.Most of the existing studies usually allocate resources evenly among tasks or adopt rule-based strategies,which cannot effectively handle dynamic MEC environments.This may cause excessive resource consumption,leading to a decrease in Quality of Service(QoS).To solve the above important problems,a Task Scheduling method with Actor-Critic(TSAC)deep reinforcement learning in MEC was proposed.First,a task scheduling model for edge environments was proposed,with task response time and task completion rate as optimization objectives.Next,based on the proposed system model and deep reinforcement learning framework,the joint optimization problem was formalized as a Markov decision process.Finally,based on the proximal policy optimization method,a novel masking mechanism was designed to improve the convergence performance of the TSAC method while avoiding agents from taking actions that violated system constraints and policy mutations.Using the real-world running datasets of the Google cluster,simulation experiments were conducted.Compared with deep Q network method,the TSAC method can decrease the average wait time by at least 6%while enhancing the task completion rate by 4%.The feasibility and effectiveness of TSAC were verified.

关 键 词：移动边缘计算 任务调度 深度强化学习 掩码机制 多目标优化 

分 类 号：TP393.17[自动化与计算机技术—计算机应用技术]