一种基于后缀数组的无词典分词方法  

An automatic and dictionary-free Chinese word segmentation method based on suffix array

作　　者：张长利[1] 赫枫龄[1] 左万利[1] 

机构地区：[1]吉林大学计算机科学与技术学院,长春130012

出　　处：《吉林大学学报（理学版）》2004年第4期548-553,共6页Journal of Jilin University:Science Edition

基　　金：国家自然科学基金(批准号:60373099).

摘　　要：提出一种基于后缀数组的无词典分词算法.该算法通过后缀数组和利用散列表获得汉字的结合模式,通过置信度筛选词.实验表明,在无需词典和语料库的前提下,该算法能够快速准确地抽取文档中的中、高频词.适用于对词条频度敏感、对计算速度要求高的中文信息处理.An automatic and dictionary-free Chinese word segmentation method based on suffix array algorithm is proposed. By the algorithm based on suffix array and by using HashMap the co-occurrence patterns of (Chinese) characters are gotten, and Chinese words are filtered through confidence. Experiment results show that by the algorithm one can acquire high frequency lexical items effectively and efficiently without the help of the dictionary and corpus as well. This method is particularly suitable for lexical-frequency-sensitive as well as time-critical Chinese information processing application.

关 键 词：无词典分词 数组 文档 散列表 计算速度 算法 中文信息处理 法能 频度 实验 

分 类 号：N02[自然科学总论—科学技术哲学] TP311.1[自动化与计算机技术—计算机软件与理论]