基于跨语言词嵌入对齐增强少数民族预训练语言模型  

Enhancing Chinese Minority Pre-trained Language Model with Cross-lingualEmbedding Alignment

作　　者：申影利 鲍薇 赵小兵[3,4] 周毛克 SHEN Yingli;BAO Wei;ZHAO Xiaobing;ZHOU Maoke(School of Chinese Ethnic Minority Languages and Literatures,Minzu University of China,Beijing 100081,China;China Electronics Standardization Institute,Beijing 100007,China;School of Information Engineering,Minzu University of China,Beijing 100081,China;National Language Resource Monitoring and Research Center of Minority Languages,Beijing 100081,China)

机构地区：[1]中央民族大学中国少数民族语言文学学院,北京100081 [2]中国电子技术标准化研究院,北京100007 [3]中央民族大学信息工程学院,北京100081 [4]国家语言资源监测与研究少数民族语言中心,北京100081

出　　处：《中文信息学报》2023年第9期63-72,共10页Journal of Chinese Information Processing

基　　金：国家社会科学基金(22&ZD035)。

摘　　要：由于民族语言与汉语之间的词嵌入语义空间差距较大,导致预训练语言模型的跨语言迁移效果不佳。为此,该文设计了一个通过静态词嵌入对齐到上下文词嵌入空间中的新框架,以提升少数民族跨语言预训练语言模型在下游任务中的表现。该文首先将由大规模单语数据训练的静态词嵌入进行跨语言对齐;其次,基于民汉平行语料从少数民族预训练语言模型CINO中抽取上下文词嵌入,并设计两种损失函数:双语词典归纳损失、对比学习损失,实现静态词嵌入对齐到上下文词嵌入的共享语义空间中;最后,我们将结合静态与上下文跨语言词嵌入对齐的CINO增强模型应用于双语词典归纳、文本分类以及机器翻译任务中,在多个语言对上的实验结果表明,相比鲁棒的基线系统,该文方法在标注语料匮乏的下游任务中均达到了显著的性能提升。The significant difference in the semantic space of word embedding between Chinese minority languages and Chinese leads to poor cross-lingual transfer of pre-trained language models.In this paper,we design a new framework for improving the performance of pre-trained language models in downstream tasks by aligning static word embeddings into contextual word embeddings space.Specifically,we first perform cross-linguistic alignment of static word embeddings trained on large-scale monolingual data.Then we extract the contextual word embeddings from CINO(Chinese Minority Pretrained Language Model)through minority language and Chinese parallel corpus.We design two loss functions:bilingual lexicon induction loss and contrast learning loss,to align the static word embeddings into the semantic space of contextual word embeddings.Finally,we apply the CINO enhanced model based on cross-lingual embedding alignment to downstream tasks such as bilingual lexicon induction,text classification,and machine translation.Experiments on multiple language pairs show that our proposed approach achieves significant improvements over robust baseline systems with limited annotation corpus.

关 键 词：词嵌入对齐 少数民族预训练语言模型 双语词典归纳 对比学习 

分 类 号：TP391[自动化与计算机技术—计算机应用技术]