基于新词发现的古典文学作品分词方法研究  

Study on Word Segmentation Method of Classical Literature Based on New Word Discovery

作　　者：高嘉琦 赵庆聪[1,2] GAO Jia-qi;ZHAO Qing-cong(School of Information Management,Beijing Information Science and Technology University,Beijing 100192,China;Beijing Key Laboratory of Big Data Decision for Green Development,Beijing 100192,China)

机构地区：[1]北京信息科技大学信息管理学院,北京100192 [2]绿色发展大数据决策北京市重点实验室,北京100192

出　　处：《计算机技术与发展》2021年第9期178-181,207,共5页Computer Technology and Development

基　　金：国家重点研发计划项目(2017YFB1400400)。

摘　　要：对于中文文本的分词研究来说,现有的分词方法和技术较多都是针对现代汉语,现代汉语的分词方法和体系已经很成熟,但对古代汉语的研究较少。由于古文的特殊性,将现代汉语的分词方法技术直接用于古汉语时,无法得到分词准确的理想效果,目前对古汉语分词方法的研究还未形成成熟的体系。文中提出一种基于新词发现的古典文学作品分词方法,即从大量古典文学作品语料中发现新词,构建古汉语分词词典,在此基础上再对古文文本进行分词。以《三国演义》古文文本处理为例,验证了基于新词发现的古典文学作品分词方法能有效提高古文分词的准确率.For the research on word segmentation of Chinese text,most of the existing word segmentation methods and technologies are aimed at modern Chinese.The word segmentation methods and systems of modern Chinese have been quite mature,but there are few studies on ancient Chinese.Due to the particularity of ancient Chinese,when the modern Chinese word segmentation method and technology are directly used in ancient Chinese,the ideal effect of accurate word segmentation cannot be obtained.At present,the word segmentation of ancient Chinese has not yet formed a general method and mature system.We propose a method of word segmentation in classical literature based on neologism discovery,that is,discovering new words from a large number of classical literary works,constructing an ancient Chinese word segmentation dictionary,and then segmenting the ancient text on this basis.Taking the ancient text processing of"The Romance of the Three Kingdoms"as an example,it is verified that the word segmentation method of classical literary works based on the discovery of new words can effectively improve the accuracy of ancient text segmentation.

关 键 词：古典文学 新词发现 分词 互信息 左右熵 

分 类 号：TP301[自动化与计算机技术—计算机系统结构]