中文文本同频词统计规律及在关键词提取中的应用  

Statistics law of same frequency words in Chinese text and its application to keywords extraction

作　　者：李晓超[1,2,3] 赵书良[1,2,3] 罗燕[1,2,3] 陈敏[1,2,3] 柳萌萌[1,2,3] 

机构地区：[1]河北师范大学数学与信息科学学院,石家庄050024 [2]河北师范大学河北省计算数学与应用重点实验室,石家庄050024 [3]河北师范大学移动物联网研究院,石家庄050024

出　　处：《计算机应用研究》2016年第4期1007-1012,共6页Application Research of Computers

基　　金：国家自然科学基金资助项目(71271067);国家社会科学基金资助项目(13BTY011);国家社科基金重大项目(13&ZD091);河北师范大学数学与信息科学学院硕士基金资助项目(201301)

摘　　要：通过大量实验对中文文本中同频词的统计规律进行了研究,利用齐普夫定律推导出了适合中文文本的同频词数的数学表达式,能更准确地表示出不同长度的文本中各频次的同频词数;借助同频词数的数学表达式,重新确立了中文文本中高频词和低频词的界分公式,并通过实验验证了该公式能够更好地界分高频词和低频词。将提出的统计规律应用于中文文本关键词提取,有效提高了关键词提取效率,在文本长度不小于3 010词的前提下,频次为1和频次为2的词不必参与TF-IDF值的计算,可将计算效率提高2~7倍,且没有造成关键词丢失。解决了学术界关心的如何处理中文低频词的问题,对关键词提取中如何处理低频词提供了可操作标准。This paper presented a statistics law on the same frequency words in Chinese text based on a large number of experiments. It deduced the mathematical expression of the same frequency words based on Zipf＇s law,which could be applied to Chinese text better. Moreover,it re-established the boundary points formula of high-frequency words and low-frequency words,and then verified its correctness. Finally,it applied the proposed statistics law to keywords extraction. Previous academic research on how to deal with low-frequency words was rare and nobody gave a concrete solution. This paper provided a standard method on how to deal with the low-frequency words in the application of keywords extraction. It notes that text length must be no less than 3 010 words and it can ignore the calculation of words occurring once and twice when calculating the value of TFIDF. This method raises the efficiency by 2 ~ 7 times but no loss of keywords.

关 键 词：同频词 齐普夫定律 布茨定律 关键词提取 TF-IDF算法 

分 类 号：TP391.1[自动化与计算机技术—计算机应用技术]