Nutch中文分词的设计与实现  

Design and Implementation of Chinese Word Segmentation in Nutch

作　　者：张敏[1] 杜华[1] 

机构地区：[1]河北北方学院信息科学与工程学院,河北张家口075000

出　　处：《河北北方学院学报（自然科学版）》2010年第4期53-56,62,共5页Journal of Hebei North University:Natural Science Edition

基　　金：河北省张家口市2009年科技攻关计划项目(0921047B)

摘　　要：面对与日俱增的中文信息检索需求,Nutch作为一个开源的搜索引擎系统平台受到众多开发者的青睐.但由于Nutch是基于英文的系统,不具备中文分词能力,因此,研究中文分词方法在Nutch中的应用对实现中文搜索引擎具有非常现实的意义.在对中文分词技术进行研究的基础上,设计并实现了具有中文分词功能和新词识别功能的分词器,在Nutch中实现了中文分词功能.实验测试结果表明,算法的分词效果能够达到预期的中文分词的要求.Faced with growing demand of Chinese information retrieval, Nutch as a platform of open source search engine is favored by many developers. Nutch is based on the English system, without the capacity of Chinese word segmentation, so it has a very practical significance for realizing the Chinese search engine to research Chinese words in the application of Nutch. Based on the study of Chinese word segmentation, a word-dividing machine with the function of Chinese word segmentation and recognizing new word, is designed and implemented for the function of Chinese word segmentation in Nutch. Experi- mental results show that the effect of the segmentation algorithm can achieve the desired Chinese word re- quirements.

关 键 词：搜索引擎 中文分词 分词器 NUTCH 

分 类 号：TP391.1[自动化与计算机技术—计算机应用技术]