基于共现词映射的中英韩跨语种文档相似度计算  

Cross-lingual Documents Similarity Measure Based on Co-occurrence Mapping Between Chinese,English and Korean

作　　者：刘娇[1] 崔荣一[1] 赵亚慧[1] LIU Jiao;CUI Rongyi;ZHAO Yahui(Intelligent Information Processing Lab. , Department of Computer Science Technology, Yanbian University, Yanji, Jilin 133002, China)

机构地区：[1]延边大学计算机科学与技术学院智能信息处理研究室,吉林延吉133002

出　　处：《中文信息学报》2018年第3期55-63,共9页Journal of Chinese Information Processing

基　　金：国家语委2015年度科研立项项目(教语信司函[2015]21号)

摘　　要：该文采用中英韩跨语种文本数据研究不同语种文档间相似度的计算方法。首先,通过共现词映射将某语种空间中的文档向量表示成另一语种空间中的文档向量;其次,利用潜在语义分析补充了不同语言间一词多义现象造成的向量缺失;最后,在具有等价语义信息的同一语种空间中计算了两个文档之间的余弦相似度。该文工作避开了外部词典和知识库,利用中英韩三个语种的对齐语料库,建立了不同语种词汇间的对应关系。结果表明,共现词映射对计算不同语种文档之间的相似度具有较大影响,对同语义的不同语种文档(即译文)的检索准确率达到95%,验证了该方法的有效性。The paper analyses the cross lingual document similarity measure between different languages,including Chinese,English,and Korean.Initially,this paper maps a document vector in a language to another by co-occurrence information.The Latent Semantic Analysis is then employed to remedy the lack caused by polysemy across languages.Finally,the cosine similarity between two documents is calculated in the same space with equivalent semantic information.This method does not rely on a pre-existing external dictionary and knowledge base,but use the parallel corpus to establish the lexical relationship between Chinese,English,and Korean.It turns out that co-occurrence mapping contributes substantially to documents similarity measure,resulting an 95% accuracy of translation retrieval.

关 键 词：跨语种文档相似度 共现词 潜在语义分析 

分 类 号：TP391[自动化与计算机技术—计算机应用技术]