基于SAC强化学习的车联网频谱资源动态分配  

Dynamic Spectrum Resource Allocation in Internet of Vehicles Based on SAC Reinforcement Learning

作　　者：黄煜梵 彭诺蘅 林艳 范建存[2] 张一晋[1] 余妍秋 HUANG Yufan;PENG Nuoheng;LIN Yan;FAN Jiancun;ZHANG Yijin;YU Yanqiu(School of Electronic and Optical Engineering,Nanjing University of Science and Technology,Nanjing 210094,China;School of Information and Communications Engineering,Xi'an Jiaotong University,Xi'an 710049,China)

机构地区：[1]南京理工大学电子工程与光电技术学院,南京210094 [2]西安交通大学信息与通信工程学院,西安710049

出　　处：《计算机工程》2021年第9期34-43,共10页Computer Engineering

基　　金：国家自然科学基金(62001225,62071236);中央高校基本科研业务费专项资金(30920021127,30919011227);江苏省自然科学青年基金(BK20190454)。

摘　　要：针对车联网频谱资源稀缺问题,提出一种基于柔性致动-评价(SAC)强化学习算法的多智能体频谱资源动态分配方案。以最大化信道总容量与载荷成功交付率为目标,建立车辆-车辆(V2V)链路频谱资源分配模型。将每条V2V链路作为单个智能体,构建多智能体马尔科夫决策过程模型。利用SAC强化学习算法设计神经网络,通过最大化熵与累计奖励和以训练智能体,使得V2V链路经过不断学习优化频谱资源分配。仿真结果表明,与基于深度Q网络和深度确定性策略梯度的频谱资源分配方案相比,该方案可以更高效地完成车联网链路之间的频谱共享任务,且信道传输速率和载荷成功交付率更高。To address the scarcity of spectrum resources in Internet of Vehicles(IoV),a novel multi-agent dynamic spectrum allocation solution based on Soft Actor-Critic(SAC)reinforcement learning is proposed.The solution aims to maximize the total channel capacity and the success rate of payload delivery.To achieve this goal,a spectrum resource allocation model consisting of Vehicle-to-Vehicle(V2V)links is constructed.Each V2V link is regarded as an agent to model this problem as a Markov decision process.Then the SAC reinforcement learning algorithm is used to design a neural network.The agents are trained by maximum entropy and cumulative reward,so the V2V links can optimize the allocation of spectrum resources through rounds of learning.Simulation results show that compared with spectrum resource allocation scheme based on Deep Q-Network(DQN)and Deep Deterministic Policy Gradient(DDPG),the proposed scheme can more efficiently implement spectrum sharing between V2V links,and improves the channel transmission rate and the success rate of payload delivery.

关 键 词：车联网 资源分配 多智能体强化学习 柔性致动-评价算法 频谱分配 

分 类 号：TP393.1[自动化与计算机技术—计算机应用技术]