利用N-gram和语义分析的维吾尔语文本相似性检测方法  

Uyghur text similarity detection method using N-gram and semantic analysis

作　　者：张莹[1] 亚森·艾则孜[1] 吴顺祥[2] Zhang Ying;Yasen Aizezi;Wu Shunxiang(Dept. of Information Security Engineering,Xinjiang Police College,Urumchi 830011,China;Dept. of Automation,Xiamen University,Xiamen Fujian 361005,China)

机构地区：[1]新疆警察学院信息安全工程系,乌鲁木齐830011 [2]厦门大学自动化系,福建厦门361005

出　　处：《计算机应用研究》2019年第9期2722-2725,2729,共5页Application Research of Computers

基　　金：国家自然科学基金资助项目(61762086);新疆维吾尔自治区高校科研计划立项项目(XJEDU2016S090)

摘　　要：为了实现维吾尔语文本的相似性检测,提出一种基于N-gram和语义分析的相似性检测方法。根据维吾尔语单词特征,采用了N-gram统计模型来获得词语,并根据词语在文本中的出现频率来构建词语-文本关系矩阵,并作为文本模型。采用了潜在语义分析(LSA)来获得词语及其文本之间的隐藏关联,以此解决维吾尔语词义模糊的问题,并获得准确的相似度。在包含重组和同义词替换的剽窃文本集上进行实验,结果表明该方法能够准确有效地检测出相似性。In order to detect similarities between Uighur texts, this paper proposed a similarity detection method based on N-gram and semantic analysis. Firstly, it used N-gram statistical model to obtain the words based on Uyghur word features, and constructed the word-text relation matrix according to the appearance frequency of the words in the text. Then, it adopted an LSA to obtain the hidden association between the words and their texts, so as to solve the problem of vague semantic meaning in Uyghur language and obtain exact similarity. Experiments on plagiarized text sets containing reorganization and synonym replacement show that this method can detect the similarity accurately and effectively.

关 键 词：维吾尔语 文本相似性检测 N-gram统计模型 潜在语义分析 

分 类 号：TP391[自动化与计算机技术—计算机应用技术]