基于Spark的大规模机器翻译系统研究  

Research on a large-scale machine translation system based on Spark

作　　者：赵崇俊[1] ZHAO Chongjun(Xianyang Normal University,Xianyang Shaanxi 712000,China)

机构地区：[1]咸阳师范学院,陕西咸阳712000

出　　处：《自动化与仪器仪表》2022年第3期73-77,共5页Automation & Instrumentation

基　　金：陕西省教育科学“十三五”规划2020年度课题,大学英语教学《指南》视域下教师隐性课程研究(SGH20Y1243);陕西省体育局科研常规课题,奥运会全运会视域下城市竞技赛事语言环境建设研究,课题号:2021249;陕西省哲学社会科学重大理论与现实问题研究重点项目,“一带一路”背景下师范类高校大学英语精品教材资源建设研究(2021HZ-842)。

摘　　要：为提高大规模机器翻译准确率和翻译效率,基于EM分布式训练方法,提出基于Spark的大规模机器翻译方法。首先,在机器翻译模型的基础上对并行化训练方法和机器翻译系统框架进行分析,然后提出EM分布式并行算法和框架,针对短语翻译模型和层次翻译模型训练的特点,利用单机多线程工具MGIZA++构建完整的词对齐训练方法对短语翻译模型进行训练;最后通过实验验证提出算法对翻译模型的应用效果。实验结果表明,采用Align_on_MGIZA模型后,短语翻译速度比Chaski模型快了2倍,翻译速度显著提升,训练时间更短,仅用了该算法可以适用于短语翻译模型的高效准确翻译,说明系统性能优越,更具有效性。To improve the large-scale machine translation accuracy and translation efficiency,large-scale Spark-based machine translation methods are proposed based on the EM distributed training method.Firstly,the parallelization training method and machine translation system framework are analyzed based on the machine translation model,and then the EM distributed parallel algorithm and framework are proposed for the characteristics of phrase translation model and the single-machine multi-thread tool MGIZA++to train the phrase translation model through experimental verification.The experimental results show that the Align_on_MGIZA model is two times faster than the Chaski model,significantly improved and has shorter training time.Only this algorithm can be applied for efficient and accurate translation of the phrase translation model,indicating that the system is superior and more efficient.

关 键 词：机器翻译 词对齐 SPARK 大规模训练 并行化算法 

分 类 号：TP392[自动化与计算机技术—计算机应用技术]