基于视觉注意力机制的异步优势行动者-评论家算法  

Asynchronous Advantage Actor-Critic Algorithm with Visual Attention Mechanism

作　　者：李杰[1,2] 凌兴宏 伏玉琛[1,2] 刘全 LI Jie;LING Xing-hong;FU Yu-chen;LIU Quan(School of Computer Science and Technology,Soochow University,Suzhou,Jiangsu 215006,China;Provincial Key Laboratory for Computer Information Processing Technology,Soochow University,Suzhou,Jiangsu 215006,China;Key Laboratory of Symbolic Computation and Knowledge Engineering of Ministry of Education,Jilin University,Changchun 130012,China;Collaborative Innovation Center of Novel Software Technology and Industrialization,Nanjing 210000,China)

机构地区：[1]苏州大学计算机科学与技术学院,江苏苏州215006 [2]苏州大学江苏省计算机信息处理技术重点实验室,江苏苏州215006 [3]吉林大学符号计算与知识工程教育部重点实验室,长春130012 [4]软件新技术与产业化协同创新中心,南京210000

出　　处：《计算机科学》2019年第5期169-174,共6页Computer Science

基　　金：国家自然科学基金项目(61772355;61702055;61303108;61373094;61472262;61502323;61502329);江苏省高等学校自然科学研究重大项目(17KJA520004);吉林大学符号计算与知识工程教育部重点实验室资助项目(93K172014K04;93K172017K18);苏州市应用基础研究计划工业部分(SYG201422);苏州市民生科技项目(SS201736)资助

摘　　要：异步深度强化学习能够通过多线程技术极大地减少学习模型所需要的训练时间。然而作为异步深度强化学习的一种经典算法,异步优势行动者-评论家算法没有充分利用某些具有重要价值的区域信息,网络模型的学习效率不够理想。针对此问题,文中提出一种基于视觉注意力机制的异步优势行动者-评论家模型。该模型在传统异步优势行动者-评论家算法的基础上引入了视觉注意力机制,通过计算图像各区域点的视觉重要性值,利用回归、加权等操作得到注意力机制的上下文向量,从而使Agent将注意力集中于面积较小但更具丰富价值的图像区域,加快网络模型解码速度,更高效地学习近似最优策略。实验结果表明,与传统的异步优势行动者-评论家算法相比,该模型在基于视觉感知的决策任务上具有更好的性能表现。Asynchronous deep reinforcement learning(ADRL)can greatly reduce the training time required for learning models by adopting the multiple threading techniques.However,as an exemplary algorithm of ADRL,asynchronous advantage actor-critic(A3C)algorithm fails to completely utilize some valuable regional information,leading to unsatisfactory performance for model training.Aiming at the above problem,this paper proposed an asynchronous advantage actor-critic model with visual attention mechanism(VAM-A3C).AM-A3C integrates visual attention mechanism with traditional asynchronous advantage actor-critic algorithms.By calculating the visual importance value of each area point in the whole image compared with the traditional Cofi algorithm,and obtaining the context vector of the attention mechanism via regression function and weighting function,Agent can focus on smaller but more valuable image areas to accelerate network model decoding and to learn the approximate optimal strategy more efficiently.Experimental results show the superior performance of VAM-A3C in some decision-making tasks based on visual perception compared with the traditional asynchronous deep reinforcement learning algorithm.

关 键 词：异步深度强化学习 视觉注意力机制 行动者-评论家 异步优势行动者-评论家 

分 类 号：TP181[自动化与计算机技术—控制理论与控制工程]