网络维吾尔文判别及其文本长度下界的探讨  

Uyghur Recognition in Webpages and the Lower Bound of Text Length for Language Discrimination

作　　者：倪耀群[1,2,3] 曹鹏[1,2] 许洪波[1] 唐慧丰[3] 程学旗[1] 

机构地区：[1]中国科学院计算技术研究所,北京100190 [2]中国科学院研究生院,北京100049 [3]解放军外国语学院,河南洛阳471003

出　　处：《中文信息学报》2012年第6期109-115,共7页Journal of Chinese Information Processing

基　　金：国家自然科学基金资助项目(60903139;60873243);自然基金重点资助项目(60933005);国家863计划重点资助项目(2010AA012502;2010AA012503)

摘　　要：将维吾尔文从阿拉伯文、哈萨克文、柯尔克孜文等以阿拉伯字母为基础书写的类似文字中识别出来,是维文信息处理的基础。作者对维吾尔字符的编码优化后使用N元语法模型实现了维吾尔文的快速语种判别,准确率超过98%。经过错误分析,发现错误判别的文本主要集中在论坛和微博客中,这些文本有效字符数太少,语言特征不充分。最后作者计算了四种语言真实网络文本中的所有公共子串,并对文种判别所需要的最短字符串长度进行了分析。Distinguishing Uyghur language from similar Arabic script languages such as Arabic, Kazakh, Kirgiz, etc. is an indispensable issue in Uyghur information processing. The paper builts a n-gram based Uyghur language discrimination model over an optimized Uyghur character encoding schema for an accuracy over 98%. The analysis reveals the misestimated texts are centered around the forum posts and microblogs because of their extremely short length （often only a few words）. Thus, the paper examines all common sub-strings among tokens appeared in web texts of the four languages and probes into the minimum string length required to determine its language.

关 键 词：老维文 语种识别 最大公共子串 

分 类 号：TP391[自动化与计算机技术—计算机应用技术]