中文农业搜索引擎字符编码识别  

Character Encoding Identification of Chinese Agriculture Search Engine

作　　者：吴乃宁[1] 张太红[1] 白涛[1] 

机构地区：[1]新疆农业大学计算机与信息工程学院,乌鲁木齐830052

出　　处：《新疆农业大学学报》2014年第5期420-423,共4页Journal of Xinjiang Agricultural University

基　　金：新疆维吾尔自治区科技攻关项目(200931103)

摘　　要：针对农业网页中汉字编码标识混乱的情况,提出了一种综合运用编码规则和网页文本特征的字符编码识别模型。利用卡方检验算法,结合最小二乘多元线性回归方法,得到了基于网页文本特征的字符识别模型。实验结果显示,在适当的选取阈值(r=1,阈值=属于某一编码的字符数/网页总字符数)和文本特征数(≥65)的基础上,模型准确率达到100%,且结果稳定。The character encoding identification model comprehensively using encoding rules and Web page text was put forward in accordance with the confused conditions of character encoding identification in Chinese agriculture web page.Using the chi-square test algorithm,combining with the method of least square multivariale linear regression,the model of character identification based on Web page text feature was obtained.The experimental results showed that the accuracy of the model reached 100% and the result was stable on the basis of the appropriate threshold selected （r =1,threshold=the number of characters belonging to a coding/the total number of web page）and the text feature number （≥65）.

关 键 词：编码识别 卡方检验 多元线性回归 GB2312 Big5 

分 类 号：TP391.3[自动化与计算机技术—计算机应用技术]