递归最小二乘循环神经网络  

Recurrent Neural Networks With Recursive Least Squares

作　　者：赵杰[1] 张春元[1] 刘超 周辉[1] 欧宜贵[2] 宋淇 ZHAO Jie;ZHANG Chun-Yuan;LIU Chao;ZHOU Hui;OU Yi-Gui;SONG Qi(School of Computer Science and Technology,Hainan University,Haikou 570228;School of Science,Hainan University,Haikou 570228)

机构地区：[1]海南大学计算机科学与技术学院,海口570228 [2]海南大学理学院,海口570228

出　　处：《自动化学报》2022年第8期2050-2061,共12页Acta Automatica Sinica

基　　金：国家自然科学基金(61762032,61662019,11961018)资助。

摘　　要：针对循环神经网络(Recurrent neural networks,RNNs)一阶优化算法学习效率不高和二阶优化算法时空开销过大,提出一种新的迷你批递归最小二乘优化算法.所提算法采用非激活线性输出误差替代传统的激活输出误差反向传播,并结合加权线性最小二乘目标函数关于隐藏层线性输出的等效梯度,逐层导出RNNs参数的迷你批递归最小二乘解.相较随机梯度下降算法,所提算法只在RNNs的隐藏层和输出层分别增加了一个协方差矩阵,其时间复杂度和空间复杂度仅为随机梯度下降算法的3倍左右.此外,本文还就所提算法的遗忘因子自适应问题和过拟合问题分别给出一种解决办法.仿真结果表明,无论是对序列数据的分类问题还是预测问题,所提算法的收敛速度要优于现有主流一阶优化算法,而且在超参数的设置上具有较好的鲁棒性.In recurrent neural networks(RNNs),the first-order optimization algorithms usually converge slowly,and the second-order optimization algorithms commonly have high time and space complexities.In order to solve these problems,a new minibatch recursive least squares(RLS)optimization algorithm is proposed.Using the inactive linear output error to replace the conventional activation output error for backpropagation,together with the equivalent gradients of the weighted linear least squares objective function with respect to linear outputs of the hidden layer,the proposed algorithm derives the minibatch recursive least squares solutions of RNNs parameters layer by layer.Compared with the stochastic gradient descent algorithm,the proposed algorithm only adds one covariance matrix into each layer of RNNs,and its time and space complexities are almost three times as much.Furthermore,in order to address the adaptive problem of the forgetting factor and the overfitting problem of the proposed algorithm,two approaches are also presented,respectively,in this paper.The simulation results,on the classification and prediction problems of sequential data,show that the proposed algorithm has faster convergence speed than popular first-order optimization algorithms.In addition,the proposed algorithm also has good robustness in the selection of hyperparameters.

关 键 词：深度学习 循环神经网络 递归最小二乘 迷你批学习 优化算法 

分 类 号：TP183[自动化与计算机技术—控制理论与控制工程]