藏文音节查错纠错模型研究  

Research on Tibetan Syllable Error Detection and Correction Model

作　　者：珠杰 郑任公 拉巴顿珠 德庆卓玛 顿珠次仁 ZHU Jie;ZHENG Rengong;LHAKPA Dondrub;DEQING Zhuoma;DUNZHU Ciren(School of Information Science and Technology,Tibet University,Lhasa,Xizang 540000,China;Tibet Informationization Provincial-Ministerial Co-Construction Collaborative Innovation Center,Lhasa,Xizang 540000,China)

机构地区：[1]西藏大学信息科学技术学院,西藏拉萨540000 [2]西藏信息化省部共建协同创新中心,西藏拉萨540000

出　　处：《中文信息学报》2024年第12期116-126,共11页Journal of Chinese Information Processing

基　　金：国家自然科学基金(62406256);新一代人工智能国家科技重大专项(2022ZD0116100);西藏自治区哲学社会科学专项(23YBE25)。

摘　　要：针对藏文文本自动校对研究中缺乏高质量标注语料、鲜有纠错任务研究等问题,该文以藏文音节为单元的等长文本为研究内容,通过分析藏文文本错误类型,开展了藏文查错、纠错模型研究,该文主要贡献如下:(1)针对缺乏标注语料问题,一是提出了结合语言知识的藏文混淆集构建算法,自动建立了音似、形似和拼写错误音节的混淆集,二是根据音似、形似、动词时态、易错虚词的不同混淆集,提出了加噪算法,在等长文本中将正确音节替换为错误音节。(2)针对查错问题,提出了基于预训练模型Word2Vec和ELMo的BiGRU-Attention藏文音节查错模型。最终实验表明,使用预训练模型能有效提升藏文音节查错效果,其中ELMo-BiGRU-Attention模型的查错效果达到最佳,音节级查错F1为90.91%,句子级查错F1为83.24%。(3)针对纠错问题,提出了soft-masked+BERT的藏文音节纠错网络,效果最好的模型音节级查错F1和纠错F1分别为95.51%和90.69%,句子级查错F1和纠错F1分别为86.34%和79.77%。To address the less-touched issue of automatic proofreading of Tibetan texts,this paper studies Tibetan error detection and correction model with a focus on equal-length texts based on Tibetan syllables.For corpus construction,this paper proposes a linguistic driven Tibetan confusion set construction algorithm which establishes confusion sets of similar sounds and shapes.It also designs a noise adding algorithm based on the different confusion sets of phonetic similarity,morphological similarity,verb tense,and error prone function words.For error detection,a BiGRU-Attention Tibetan syllable error detection model based on pre-trained models Word2Vec and ELMo is applied.For error correction,a soft-masked+BERT Tibetan syllable correction network is deployed.Experiments demonstrate that the best F1-values for error checking and error correction reach 95.51%and 90.69%at the syllable level and 86.34%and 79.77%at sentence level,respectively.

关 键 词：藏文音节 查错模型 纠错模型 预训练 软掩码 

分 类 号：TP391[自动化与计算机技术—计算机应用技术]