基于长度递减与串频统计的文本切分算法  

Algorithm for Chinese Text Based on Length Descending and String Frequency Statistics

作　　者：姜韶华[1] 党延忠[1] 

机构地区：[1]大连理工大学系统工程研究所,大连116024

出　　处：《情报学报》2006年第1期74-79,共6页Journal of the China Society for Scientific and Technical Information

基　　金：国家自然科学基金资助项目（项目编号：70271046）.

摘　　要：提出了一种基于汉字串频度及串长度递减的中文文本自动切分算法。采用长串优先匹配法，不需要词典，不需要事先估计字之间的搭配概率，不需要建立字索引，利用串频信息可以自动切分出文本中有意义的汉字串。该算法能够有效地切分出文本中新涌现的通用词、专业术语及专有名词，并且能够有效避免具有包含关系的长、短汉字串中的短汉字串的错误统计。实验表明，在无需语料库学习的情况下，该算法能够快速、准确地切分出中文文档中出现频率大于等于支持度阈值的汉字串。This paper puts forward a new algorithm about automatic Chinese text segmentation based on Chinese charaeters string frequency and length descending. It can automatically distinguish meaningful Chinese characters string in text based on processing longer string first and string frequency information without the help of dictionary, without the help of acquiring the probability between words in advance and without the help of Chinese character index, This algorithm can effectively distinguish new universal words, specialized terms and proper nouns, and it can effectively avoid statistical error about the shorter string which is belonged to a longer one. Experiment results show that this algorithm can rapidly and exactly distill words which frequency is larger than predefined value without the help of text corpus studying beforehand and dictionary.

关 键 词：汉字 自动切分 串频 长串优先匹配 

分 类 号：H315.9[语言文字—英语] TP391[自动化与计算机技术—计算机应用技术]