Hadoop云平台下Nutch中文分词的研究与实现  

Research and Implementation of Nutch Chinese Word Segmentation in Hadoop Cloud Platform

作　　者：朱潜[1,2] 吴辰铌[1] 朱志良[1] 刘洪娟[1,2] 

机构地区：[1]东北大学软件学院,沈阳110819 [2]东北大学信息科学与工程学院,沈阳110819

出　　处：《小型微型计算机系统》2013年第12期2772-2776,共5页Journal of Chinese Computer Systems

基　　金：辽宁省自然科学基金项目(201202076)资助

摘　　要：Nutch是开源搜索引擎,Hadoop是Apache开发的类似于Google GFS和MapReduce的开源云平台.利用Nutch和Hadoop可以设计高效、可靠、可扩展的搜索引擎,然而Nutch的分词模块对中文进行单字切分,不符合在汉语中以词语切分的习惯,为了解决这个问题,采用词典分词的中文分词器IK Analyzer对Nutch的分词模块进行改进,首先描述在Nutch上实现IK Analyzer的方法,然后在Hadoop云环境下对该分词模块进行测试.测试结果表明,IK Analyzer的中文分词效果很符合汉语的习惯,对Nutch的理论和应用研究具有一定的意义.Nutch is an open source search engine. Hadoop which Apache develops is an open source cloud platform similar to GFS and MapReduce of Google. An efficient, reliable and scalable Chinese search engine can be designed using Nutch and Hadoop. However, the word segmentation module of Nutch segments Chinese into some single characters and doesn＇t conform to Chinese language habit of segmenting Chinese into some words. To solve this problem, this paper proposes to use IK Analyzer Chinese word segmentation toolkit based on dictionary to transform Nutch. First, the paper describes the method that how to implement IK Analyzer Chinese word segmentation module on Nutch, then tests the word segmentation module in the hadoop cloud platform. Experiments show that the word segmentation result of IK Analyzer module more conforms to Chinese language habit and has large significances on the theo- ry and application of Nutch.

关 键 词：NUTCH HADOOP 中文分词 云计算 

分 类 号：TP311[自动化与计算机技术—计算机软件与理论]