基于机器学习方法的哈萨克语词干切分研究  

Research on Kazakh Stemming Based on Machine Learning

作　　者：库瓦特拜克·马木提 Kuwatebaike·MAMUTI(School of Electronic and Information Engineering,Yili Normal University,Yining 835000,China)

机构地区：[1]伊犁师范大学电子与信息工程学院,新疆伊宁835000

出　　处：《计算机技术与发展》2020年第4期182-188,共7页Computer Technology and Development

基　　金：新疆自然科学基金(2019D01C337);伊犁师范大学科研项目(2016YSYB09);伊犁师范大学教育教学研究项目(JGZH17151)。

摘　　要：自然语言处理任务中词处理是基础性的工作,其结果直接影响后续任务的效果。词干和构形附加成分是哈萨克语单词的组成成分,其中词干显示单词的主要意义,而构形附加成分中包含着词法和句法信息,因此词干切分是对哈萨克语进行有效处理的基础。文中构建了哈萨克语词干切分语料库,并通过将哈萨克语词干切分看作是序列化标注问题,提出一种有效的哈萨克语词标注方法,并基于最大熵模型和条件随机场模型构建了对比词干切分实验。结果表明基于条件随机场模型的词干切分准确率比现有最好的哈萨克语词干切分系统的准确率有15%的提高。该方法对哈萨克语词干切分相较于基于规则的方法有了一定的提升。Word processing is a basic task in natural language processing,which directly affects the subsequent tasks.Stem and inflectional suffix are the main components of Kazakh words.Stem displays the main significance of the word,and the inflectional suffix contains lots of information of grammar and syntax.As a result,stemming becomes the basis of Kazakh information processing.We build the Kazakh segmentation corpus,and through the Kazakh stemming as serialized label problem,propose an effective Kazakh word labeling method.Based on the maximum entropy model and the conditional random field model,a comparative word-stem segmentation experiment is constructed.It is showed that the stemming accuracy based on conditional random field model is 15%higher than that of the best Kazakh stemming system.Compared with the rule-based method,the proposed method improves the stemming of Kazakh words.

关 键 词：词干切分 统计学习模型 最大熵模型 条件随机场模型 

分 类 号：TP391[自动化与计算机技术—计算机应用技术]