开源中文分词器在web搜索引擎中的应用  

The Application of Open Source Chinese Tokenizer in Web Search Engine

作　　者：刘晓婉[1] 胡燕祝[2] 艾新波[2] 

机构地区：[1]北京邮电大学电子工程学院,北京100876 [2]北京邮电大学自动化学院,北京100876

出　　处：《软件》2013年第3期80-83,共4页Software

摘　　要：目前,在信息检索方面,一般分词器对英文的支持相对较好,但对中文分词效果不太理想;随着web搜索在互联网中的广泛应用,中文分词器也出现了突飞猛进的发展,尤其以IKAnalyzer分词器为主的分词工具,为更多企事业单位所采纳。本文主要在词条化等方面对分词原理进行深入的分析,并且对如何构造智能分词词典进行了探讨。同时,对分词内部结构进行研究,在分词数目和速率方面对目前主流的中文分词器进行了比较,并运用htmlparser库对北京邮电大学的网站进行解析,采用分词工具Paoding Analysis建立索引文件存入本地,利用solt4.0全文搜索服务器给出运用中文分词后的查询效果图,实现对北京邮电大学网站的全文搜索功能。Currently,In the information retrieval aspects,Tokenizer support English relatively good.But not so ideal for Chinese word segmentation effect.With the wide application of web search in the Internet,the Chinese tokenizer also in the rapid development. Especially,IKAnalyzer Tokenizer have adopted by more and more enterprises.This paper mainly deep analyze the principle of word segmentation,in addition,discuss how to structure intelligent segmentation dictionary.Meanwhile,when we analysis of the internal structure of word segmentation,we give the current mainstream Chinese word segmentation comparison In the segmentation number and rate aspects.Use htmlparser library to analysis of the web site of Beijing university of posts and telecommunications and word segmentation tools---Paoding Analysis---to establish index file in local.Use solr4.0 text search server to give design sketch after apply Chinese word segmentation,realize the function of full text search at Beijing university of posts and telecommunications website.

关 键 词：分词器 WEB搜索 信息检索 智能分词 全文搜索 

分 类 号：TP391.3[自动化与计算机技术—计算机应用技术]