循环神经网络结构中激活函数的改进  

Improvement on Activation Functions of Recurrent Neural Network Architectures

作　　者：叶小舟[1] 陶飞飞 戚荣志[1] 张云飞[1] 周思琪[1] 刘璇[1] 

机构地区：[1]河海大学计算机与信息学院,江苏南京210098

出　　处：《计算机与现代化》2016年第12期29-33,共5页Computer and Modernization

基　　金：国家科技支撑计划项目(2013BAB06B04;HNKJ13-H17-04);国家自然科学基金面上资助项目(61272543);水利部公益性行业科研专项重点项目(201501007);NSFC-广东联合基金重点项目(U1301252)

摘　　要：循环神经网络相比于其他深度学习网络,优势在于可以学习到长时依赖知识,但学习过程中的梯度消失和爆炸问题严重阻碍了知识的按序传播,导致长时依赖知识的学习结果出现偏差。为此,已有研究主要对经典循环神经网络的结构进行改进以解决此类问题。本文分析2种类型的激活函数对传统RNN和包含门机制RNN的影响,在传统RNN结构的基础上提出改进后的模型,同时对LSTM和GRU模型的门机制进行改进。以PTB经典文本数据集和LMRD情感分类数据集进行实验,结果表明改进后的模型优于传统模型,能够有效提升模型的学习能力。Recurrent neural network has the advantage of learning long term dependencies, in contrast with other deep learning network architectures. However, the problems of vanishing and exploding gradients seriously obstruct the transmission of information over time, resulting in the deviation of learning long term dependencies. Hence, a great deal of studies focus on the adaption of classical recurrent neural network architectures. In this paper, we analyse the effect of two types of activation function for basic RNN and RNNs with gating mechanism. An improved model based on the basic RNN structure is proposed. The improved gating mechanisms of LSTM model and GRU model are proposed. Experiments on PTB classical dataset LMRD feeling classified dataset show that the improved models are advanced than traditional models and greatly improve the learning ability of the models.

关 键 词：深度学习 循环神经网络 激活函数 LSTM模型 GRU模型 

分 类 号：TP311[自动化与计算机技术—计算机软件与理论]