深度强化学习在典型网络系统中的应用综述  

A Survey on the Applications of Deep Reinforcement Learning in Classical Networking Systems

作　　者：郑莹 段庆洋 林利祥 游新宇 徐跃东 王新[1,3] ZHENG Ying;DUAN Qingyang;LIN Lixiang;YOU Xinyu;XU Yuedong;WANG Xin(School of Computer Science,Fudan University,Shanghai 200433,China;School of Information Science and Engineering,Fudan University,Shanghai 200433,China;Shanghai Key Laboratory of Intelligent Information Processing,Shanghai 200433,China)

机构地区：[1]复旦大学计算机科学技术学院,上海200433 [2]复旦大学信息科学与工程学院,上海200433 [3]上海市智能信息处理重点实验室,上海200433

出　　处：《无线电通信技术》2020年第6期603-623,共21页Radio Communications Technology

基　　金：国家自然科学基金项目(61772139,61971145);上海市港澳台科技合作项目(18510760900);广东省重点领域研发计划项目(2020B010166003)。

摘　　要：近几年来,以深度强化学习(Deep Reinforcement Learning,DRL)为代表的人工智能技术被引入计算机网络系统设计中,促使网络领域走向数据驱动和智能化,并在典型的网络系统中不断取得新的突破。计算机网络应用的难点是难以对多变的网络环境进行复杂准确的建模,借助深度神经网络出色的特征提取能力,深度强化学习能够更好地以试错的方式探索更优的决策,并具有端到端的设计优势。首先阐述深度强化学习技术的原理,包括多种典型的深度学习中使用的神经网络结构、基于值函数和基于策略梯度的深度强化学习训练算法;之后详细分析了深度强化学习技术在计算机网络领域中解决资源调度问题的研究现状,包括任务调度、视频传输、路由选择、TCP拥塞控制以及网络缓存;最后给出了在计算机网络应用中使用深度强化学习仍存在的挑战。In the past few years,Artificial Intelligence technology represented by deep reinforcement learning(DRL)has been introduced into the design of computer network systems,propelling the networking research towards a data-driven and intelligent paradigm.New breakthroughs have been continuously made in typical networking systems.The common challenge of versatile networking applications is that they are difficult to be modelled in complex network environments.With the excellent feature extraction capability of deep neural network,deep reinforcement learning is able to perform better decision-makings through trial-and-errors.Meanwhile,it demonstrates the privilege of an end-to-end design rationale.This article first explains the principles of deep reinforcement learning,including a variety of typical neural network structures,training algorithms of deep reinforcement learning based on value functions and policy gradients.We then describe in details the deep reinforcement learning applications in networking systems consisting of job scheduling,video streaming,routing,TCP congestion control and content caching.Finally,this article describes the challenges of using deep reinforcement learning in computer networking applications.

关 键 词：深度强化学习 计算机网络 任务调度 视频传输 路由选择 TCP拥塞控制 网络缓存 

分 类 号：TP393[自动化与计算机技术—计算机应用技术] TP181[自动化与计算机技术—计算机科学与技术]