藏文自动分词技术研究  

Research of Tibetan Automatic Word Segmentation

作　　者：刘晓彤 赵小兵[1,2] LIU Xiaotong;ZHAO Xiaobing(Information Engineering Institute,Minzu University of China,Beijing 100081;National Language Resource Monitoring&Research Center of Minority Languages,Minzu University of China,Beijing 100081,China)

机构地区：[1]中央民族大学信息工程学院,北京100081 [2]中央民族大学国家语言资源监测与研究少数民族语言中心,北京100081

出　　处：《中央民族大学学报（自然科学版）》2022年第2期63-66,共4页Journal of Minzu University of China(Natural Sciences Edition)

基　　金：国家语委中心项目“多民族语言文本敏感信息监测与预警技术研究”(ZDI135-98)。

摘　　要：分词是自然语言处理的基础任务,本文针对藏文分词中的特殊问题及其研究成果进行了梳理,为今后学者开展该领域的相关研究提供参考和借鉴。重点阐述和分析了藏文自身语法带来的分词难点和藏文分词时的歧义和未登录词问题,并分析了目前藏文领域研究的最新方法和发展趋势。结果表明,藏文分词的重难点在各种方法中得到了一定程度的解决,但各种单一方法都存在不足,必要时可多种方法结合使用。Word segmentation is a basic task of natural language processing. This paper summarized the problems in Tibetan word segmentation and research progress, so as to provided reference for further research.The difficulties of word segmentation brought about by Tibetan grammar and the ambiguity and out-of-vocabulary in Tibetan word segmentation were analyzed. Then analyzes and looks forward to the latest methods and development trends in the field.The Rey points of Tibetan word segmentation have been solved by various methods in a certain way, but each of them has shortcomings, multiple methods can be used in combination if necessary.

关 键 词：藏文分词 语法 未登录词 歧义 

分 类 号：G350[文化科学—情报学]