基于层级化数据记忆池的边缘侧半监督持续学习方法  

Hierarchical Memory Pool Based Edge Semi-supervised Continual Learning Method

作　　者：王祥炜 韩锐 刘驰 WANG Xiangwei;HAN Rui;Chi Harold LIU(School of Computer Science and Technology,Beijing Institute of Technology,Beijing 100081,China)

机构地区：[1]北京理工大学计算机学院,北京100081

出　　处：《计算机科学》2023年第2期23-31,共9页Computer Science

基　　金：国家自然科学基金(62132019,62272046,61872337)。

摘　　要：外界环境的不断变化导致基于传统深度学习方法的神经网络性能有不同程度的下降,因此持续学习技术逐渐受到了越来越多研究人员的关注。在边缘侧环境下,面向边缘智能的持续学习模型不仅需要解决灾难性遗忘问题,还需要面对资源严重受限这一巨大挑战。这一挑战主要体现在两个方面:1)难以在短时间内花费较大的人工开销进行样本标注,导致有标注样本资源不足;2)难以在边缘平台部署大量高算力设备,导致设备资源十分有限。然而,面对这些挑战,一方面,现有经典的持续学习方法通常需要大量有标注样本才能维护模型的可塑性与稳定性,标注资源的缺乏将导致其准确率明显下降;另一方面,为了应对标注资源不足的问题,半监督学习方法为了达到更高的模型准确率,往往需要付出较大的计算开销。针对这些问题,提出了一个面向边缘侧的,能够有效利用大量无标注样本及少量有标注样本的低开销的半监督持续学习方法(Edge Hierarchical Memory Learner,简称为EdgeHML)。EdgeHML通过构建层级化数据记忆池,使用多层存储结构对学习过程中的样本进行分级保存及回放,以在线与离线相结合的策略实现不同层级间的交互,帮助模型在半监督持续学习环境下学习新知识的同时更有效地回忆旧知识。同时,为了进一步降低针对无标注样本的计算开销,EdgeHML在记忆池的基础上,引入了渐进式学习的方法,通过控制模型对无标注样本的学习过程来减少无标注样本的迭代周期。实验结果表明,在CIFAR-10,CIFAR-100以及TinyImageNet这3种不同规模的数据集构建的半监督持续学习任务上,EdgeHML相比经典的持续学习方法,在标注资源严重受限的条件下最高提升了约16.35%的模型准确率;相比半监督持续学习方法,在保证模型性能的条件下最高缩短了超过50%的训练迭代时间,实现了边缘侧高性能、低开销The continuous changes of the external environment lead to the performance regression of neural networksbased on traditional deep learning methods.Therefore,continual learning(CL)area gradually attracts the attention of more researchers.For edge intelligence,the CL model not only needs to overcome catastrophic forgetting,but also needs to face the huge challenge of severely limited resources.This challenge is mainly reflected in the lack of labeled resources and powerful devices.However,the existing classic CL methods usually rely on a large number of labeled samples to maintain the plasticity and stability,and the lack of labeled resources will lead to a significant accuracy drop.Meanwhile,in order to deal with the problem of insufficient annotation resources,semi-supervised learning methods often need to pay a large computational and memory overhead for higher accuracy.In response to these problems,a low-cost semi-supervised CL method named edge hierarchicalmemory learner(EdgeHML)is proposed.EdgeHML can effectively utilize a large number of unlabeled samples and a small number of labeled samples.It is based on a hierarchical memory pool,leverage multi-level storage structure to store and replay samples.EdgeHML implements the interaction between different levels through a combination of online and offline strategies.In addition,in order to further reduce the computational overhead for unlabeled samples,EdgeHML leverages a progressive learning method.It reduces the computation cycles of unlabeled samples by controlling the learning process.Experimental results show that on three semi-supervised CL tasks,EdgeHML can improve the model accuracy by up to 16.35%compared with the classic CL method,and the training iterations time can be reduced by more than 50%compared with semi-supervised methods.EdgeHML achieves a semi-supervised CL process with high performance and low overhead for edge intelligence.

关 键 词：边缘智能 持续学习 半监督学习 数据标注 深度神经网络 

分 类 号：TP301[自动化与计算机技术—计算机系统结构]