基于长短时记忆和条件随机场藏文分词模型  

Tibetan Word Segmentation Model Based on LSTM and CRF

作　　者：于永斌[1] 陆瑞军 尼玛扎西[2] 群诺[2] 王昊[1] 唐倩 彭辰辉 项秀才让 YU Yongbin;LU Ruijun;NYIMA Tashi;QUN Nuo;WANG Hao;TANG Qian;PENG Chenhui;XIANGXIU Cairang(University of Electronic Science and Technology of China,Sichuan 610054,China;Tibet University,Tibet 850000,China)

机构地区：[1]电子科技大学,成都610054 [2]西藏大学,拉萨850000

出　　处：《情报工程》2023年第2期108-116,共9页Technology Intelligence Engineering

基　　金：科技创新2030-“新一代人工智能”重大项目-藏语言文字自动识别技术研发和应用(项目编号:2022ZD0116100)。

摘　　要：[目的/意义]本文提出基于长短时记忆(Long short-term memory,LSTM)神经网络和条件随机场(Conditional Random Field,CRF)的藏文分词模型。[方法/过程]引入注意力机制,获取更多特征信息,提升模型关注上下文信息与当前音节之间联系;提出一种音节扩展方法,获取更多的输入特征信息与语料信息,增强模型单音节特征信息以获取更多语义信息的能力。[局限]本文在西藏大学数据集12261条的基础上,扩充至74384条,形成Tibetan-News数据集。[结果/结论]实验结果表明,在模型中加入注意力机制并使用音节扩展方法后,模型在Tibetan-News数据集上的精确率、召回率和F1分别提升2.9%、3.5%和3.2%。基于本文模型的分词系统已在工程上应用推广。[Objective/Significance]This paper proposes a deep recurrent neural network Tibetan word separation model based on Long short-term memory(LSTM)and Conditional Random Field(CRF).[Methods/Processes]The soft attention mechanism is applied to improve the ability to extract the context information of Tibetan text sequences,and the syllable expansion method is applied to improve the single syllable and semantic feature.[Limitations]Based on the Tibetan University’s dataset,this paper constructs the Tibetan-News dataset from 12261 sentences to 74384 sentences.[Results/Conclusions]The experimental results 109 show that,compared with the Tibetan word segmentation models of LSTM and CRF,the accuracy,recall and F1 of the Tibetan word segmentation models based on soft attention LSTM and CRF on Tibet-News dataset are respectively Up 2.9%,3.5%and 3.2%.The segmentation system based on this paper is already applied in engineering field.

关 键 词：藏文分词 长短时记忆网络 条件随机场 注意力机制 

分 类 号：G35[文化科学—情报学] TP391[自动化与计算机技术—计算机应用技术]