混合语料的新词识别算法设计  

Design of New Word Recognition Algorithm for Mixed Corpus

作　　者：余小燕[1] YU Xiaoyan(College of Mathematics and Computer Science,Fuzhou University,Fuzhou,China,350108)

机构地区：[1]福州大学数学与计算机科学学院,福州350108

出　　处：《福建电脑》2021年第4期1-5,共5页Journal of Fujian Computer

基　　金：福建省中青年教师教育科研资助项目(No.JAT170102)资助。

摘　　要：随着全球化趋势和国际交流的日益频繁,语言之间的渗透与融合日渐增加,不同语种夹杂的表达方式在生活中也相当普遍,特别是在使用双语或多语的地区。作为一个多民族融合的国家,我国各地的方言与普通话夹杂的表达方式也不在少数。这种语言混用的现象造成了分词识别上的困难。本文收集了各种多语夹杂的相关语料,分析了多语夹杂的语言特征,在此基础上提出了以互信息(mutual information)和熵(entropy)过滤为基础的新词识别算法,有效提高了从混合语料中识别新词的精度。With the trend of globalization and the increasing frequency of international communication, the penetration and integration of languages are increasing day by day. The expression of mixed languages is quite common in our life, especially in areas where bilingual or multilingual languages are used. As a multi-ethnic country, there are also a few expressions mixed with Mandarin in local dialects, which is a phenomenon of mixed language.This paper collects relevant corpus of various multilingual inclusions and analyses the linguistic characteristics of multilingual inclusions. Based on this, a new word recognition algorithm based on mutual information and Entropy filtering is proposed, which effectively improves the accuracy of recognizing new words from mixed corpus.

关 键 词：混合语料 新词识别 互信息 熵 

分 类 号：TP391[自动化与计算机技术—计算机应用技术]