基于规则和统计的日语分词和词性标注的研究  

Study on Japanese Word Segmentation and POS Tagging Based on Rules and Statistics

作　　者：姜尚仆[1,2] 陈群秀[1,2] 

机构地区：[1]清华大学信息科学与技术国家实验室,北京100084 [2]清华大学计算机科学与技术系,北京100084

出　　处：《中文信息学报》2010年第1期117-122,共6页Journal of Chinese Information Processing

基　　金：国家863计划重点资助项目(2006AA010109)

摘　　要：日语分词和词性标注是以日语为源语言的机器翻译等自然语言处理工作的第一步。该文提出了一种基于规则和统计的日语分词和词性标注方法,使用基于单一感知器的联合分词和词性标注算法作为基本框架,在其中加入了基于规则的词语的邻接属性作为特征。在小规模测试集上的实验结果表明,这种方法分词的F值达到了98.2%,分词加词性标注的F值达到了94.8%。该文所采用的方法已经成功应用到日汉机器翻译系统中。Word segmentation and part-of-speech tagging is the first step of Japanese natural language processing tasks, such as machine translation in which Japanese is the source language. In this paper, a Japanese word segmentation and POS tagging approach based on rules and statistics is proposed. Adopting a single perceptron based joint word segmentation and POS tagging algorithm as the basic framework, this method is combined with the features of adjacency attributes which are derived by heuristic rules. The experiment on a small test dataset shows that the new approach achieves an F-score of 98.2G on word segmentation, and 94.8% on both word segmentation and POS tagging. This work has already been applied into the Japanese-Chinese machine translation system successfully.

关 键 词：人工智能 机器翻译 日汉机器翻译系统 日语分词 日语词性标注 联合分词 

分 类 号：TP391[自动化与计算机技术—计算机应用技术]