支持鹏程系列开源大模型应用生态演化的可持续学习能力探索  

Exploration of the continual learning ability that supports the application ecological evolution of the large-scale pretraining Peng Cheng series open source models

作　　者：余跃 刘欣[1] 蒋芳清 张晗 王晖 曾炜 YU Yue;LIU Xin;JIANG Fangqing;ZHANG Han;WANG Hui;ZENG Wei(Open Source Institution,Network Intelligence Department,Peng Cheng Laboratory,Shenzhen 518055,China;National University of Defense Technology,Changsha 410073,China;Peking University,Beijing 100091,China)

机构地区：[1]鹏城实验室网络智能部开源所,广东深圳518055 [2]国防科技大学,湖南长沙410073 [3]北京大学,北京100091

出　　处：《智能科学与技术学报》2022年第1期97-108,共12页Chinese Journal of Intelligent Science and Technology

基　　金：新形势下我国技术开源战略研究(NO.GHZX2020ZCQ013)。

摘　　要：大规模预训练模型利用大规模语料以及多样化的预训练任务在自然语言处理领域取得了巨大的成功。随着大模型的逐步发展,大模型的可持续学习能力探索成为新的研究热点。主要介绍鹏程系列大模型持续学习的技术体系、应用实践以及面临的挑战,包括通过任务扩展、数据增量和知识推理的鹏程系列可持续学习技术体系,开源大模型鹏程·盘古多任务可持续学习和鹏程·通言大模型的可持续学习能力实践,大模型可持续学习过程中面临的词表更新、语义映射和知识冲突等挑战。Large-scale pre-training models have achieved great success in the field of natural language processing by us-ing large-scale corpora and pre-training tasks.With the gradual development of large models,the continual learning abil-ity of large models has become a new research focus.The continual learning technology of the Peng Cheng series large models,the exploration of practice and the still facing challenges were mainly introduced,including the Peng Cheng se-ries continual learning technology through task expansion,data increment and knowledge reasoning,Peng Cheng PAN-GU multi-task continual learning and the practical exploration of the continual learning ability of the Peng Cheng TON-GYAN open source large model,the vocabulary update,semantic mapping and knowledge conflicts that the large model faces in the process of continual learning.

关 键 词：鹏程系列大模型 可持续学习 鹏程·盘古 鹏程·通言 开源大模型 

分 类 号：TP391.1[自动化与计算机技术—计算机应用技术]