一种基于二元模型的分层文本过滤方法  

A LAYERED TEXT FILTERING METHOD BASED ON BIGRAM MODEL

作　　者：周聚[1,2] 李培峰[1,2] 朱巧明[2] 

机构地区：[1]苏州大学计算机科学与技术学院,江苏苏州215006 [2]苏州大学江苏省计算机信息处理技术重点实验室,江苏苏州215006

出　　处：《计算机应用与软件》2011年第7期16-18,共3页Computer Applications and Software

基　　金：国家自然科学基金(90920004;60970056;60873150);江苏省自然科学基金(BK2008160);江苏省高校自然科学重大基础研究项目(08KJA520002)

摘　　要：提出一种基于二元模型的分层过滤策略的中文文本过滤方法。首先,在非法文本集中使用文档频率和卡方统计相结合的方法抽取非法关键词集合,并根据制定的策略,筛选出非法文本和一些包含非法关键词的合法文本;其次,在筛选出的文本中,选取包含非法关键词的二元词串作为特征集合,以卡方统计方法对特征进行评估,选取预定数目作为结果的特征子集,使用支持向量机分类器过滤非法文本。实验表明提出的过滤方法的准确率、召唤率和F1的值分别为:95.65%,84.87%和89.93%。This paper presents a Chinese text filtering method that uses layered filtering strategy based on bigram model.First,it extracts illegal keywords set from illegal text collection by using a method integrating the document frequency and the chi-square statistics,and then screens out illegal texts and some legal texts including illegal keywords according to the strategy set down.Secondly,it collects all bigram strings which include illegal keywords as features collection from the sifted texts,and then selects a predetermined number of the features from that collection as the resulting feature subset after assessing these features with chi-square statistics.Finally,it filters the illegal text by SVM classifier.Experimental results show that our method has achieved the precision rate,recall rate and the value of F1 to be 95.65%,84.87% and 89.93% respectively.

关 键 词：文本过滤 卡方统计 关键词抽取 二元词串 

分 类 号：TP391[自动化与计算机技术—计算机应用技术]