基于熵的机器翻译伪并行语料库选择方法  

Entropy-Based Pseudo-Parallel Corpus Selection Method for Machine Translation

作　　者：刘婉月 艾山・吾买尔 敖乃翔 郭锐 LIU Wanyue;AISHAN Wumaier;AO Naixiang;GUO Rui(College of Information Science and Engineering,Xinjiang University,Urumqi 830046;Xinjiang Laboratory of Multi-Language Information Technology,Xinjiang University,Urumqi 830046;Institute of Electronic Science,China Electronics Technology Corporation,Beijing 100041;Xinjiang Lianhai Chuangzhi Information Technology Co.,Ltd.,Urumqi 830010)

机构地区：[1]新疆大学信息科学与工程学院,乌鲁木齐830046 [2]新疆大学新疆多语种信息技术实验室,乌鲁木齐830046 [3]中国电子科技集团公司电子科学研究院,北京100041 [4]新疆联海创智信息科技有限公司,乌鲁木齐830010

出　　处：《现代计算机》2021年第19期9-14,18,共7页Modern Computer

基　　金：国家自然科学基金项目(No.61662077、61262060);新疆多语种信息技术实验室开放课题(No.2016D03023)。

摘　　要：为了获得高质量的NMT模型,大规模并行语料库是必不可少的。使用反向翻译构造伪造语料库,这些伪造语料中存在大量的噪声,这些噪声会降低NMT模型的翻译质量。为了从反向翻译的伪造语料中选取高质量的语料,对此,提出了多种基于熵过滤伪造语料的方法,使用单个或多个语言模型按照不同的困惑度选择方法筛选伪造语料,实验表明,联合不同语言模型能够筛选高质量语料,并提高模型的质量。In order to obtain high-quality NMT models,large-scale parallel corpus is essential.Using back translation to construct forged corpus,there is a lot of noise in these forged corpus,which will reduce the translation quality of NMT model.In order to select high-quality corpus from back translation forged corpus,a variety of entropy-based filtering methods are proposed to filter forged corpus.Single or multiple language models are used to screen forged corpus according to different confusion selection methods.Experiments show that combining different language models can screen high-quality corpus and improve the quality of the models.

关 键 词：大规模并行语料库 语言模型 伪造语料库 困惑度 

分 类 号：TP391.2[自动化与计算机技术—计算机应用技术]