CPT:a pre-trained unbalanced transformer for both Chinese language understanding and generation  

作　　者：Yunfan SHAO Zhichao GENG Yitao LIU Junqi DAI Hang YAN Fei YANG Zhe LI Hujun BAO Xipeng QIU 

机构地区：[1]School of Computer Science,Fudan University,Shanghai 200433,China [2]Zhejiang Lab,Hangzhou 311121,China [3]Shanghai Key Laboratory of Intelligent Information Processing,Fudan University,Shanghai 200433,China

出　　处：《Science China(Information Sciences)》2024年第5期39-51,共13页中国科学（信息科学）（英文版）

基　　金：supported by National Key Research and Development Program of China(Grant No.2020AAA0108702);National Natural Science Foundation of China(Grant No.62022027).

摘　　要：In this paper,we take the advantage of previous pre-trained models(PTMs)and propose a novel Chinese pre-trained unbalanced transformer(CPT).Different from previous Chinese PTMs,CPT is designed to utilize the shared knowledge between natural language understanding(NLU)and natural language generation(NLG)to boost the performance.CPT consists of three parts:a shared encoder,an understanding decoder,and a generation decoder.Two specific decoders with a shared encoder are pretrained with masked language modeling(MLM)and denoising auto-encoding(DAE)tasks,respectively.With the partially shared architecture and multi-task pre-training,CPT can(1)learn specific knowledge of both NLU or NLG tasks with two decoders and(2)be fine-tuned flexibly that fully exploits the potential of the model.Moreover,the unbalanced transformer saves the computational and storage cost,which makes CPT competitive and greatly accelerates the inference of text generation.Experimental results on a wide range of Chinese NLU and NLG tasks show the effectiveness of CPT.

关 键 词：pre-trained model TRANSFORMER language model GENERATION unified model 

分 类 号：TP391.1[自动化与计算机技术—计算机应用技术]