大语言模型多语言词对齐能力评测方法  

Evaluation methods for multilingual word alignment capabilities in large language models

作　　者：李洁 李正芳 邹垚 熊大卫 胡建[1] LI Jie;LI Zheng-fang;ZOU Yao;XIONG Da-wei;HU Jian(School of Computer Science and Artificial Intelligence,Southwest Minzu University,Chengdu 610041,China)

机构地区：[1]西南民族大学计算机与人工智能学院,四川成都610041

出　　处：《西南民族大学学报(自然科学版)》2024年第6期681-688,共8页Journal of Southwest Minzu University(Natural Science Edition)

基　　金：国家社会科学基金重大招标项目(19ZDA284);西南民族大学研究生创新型科研项目(YCYB2024005)。

摘　　要：针对目前大语言模型多语言词对齐能力评测相对缺乏的问题,提出一种通过跨语言选词填空任务评测其多语言词对齐能力的方法:根据词汇的上下文长短、词性、干扰词数量等规则生成多维度的跨语言选词填空评测数据集,并用其对多种大语言模型的多语言词对齐能力进行评测.在以中文和英文两种语言以及代表性大语言模型为例的实验结果显示,多个大语言模型在中英跨语言选词填空任务上的准确率超过80%,最高达90.4%,证明大语言模型具有较优的多语言词对齐能力.不仅可为评测大语言模型多语言词对齐能力提供方法和测试数据,也可为多语言共同性和跨语言处理任务研究者提供模型选择建议.To solve the problem that the relative lack of current evaluations on the multilingual word alignment capabilities of Large Language Models,this paper introduced a method to evaluate these capabilities through a cross-lingual cloze.The method involved generating a multidimensional evaluation datasets based on rules such as the context length,part of speech,and number of distractor words of the vocabulary.This datasets were used to test the multilingual word alignment abilities of various Large Language Models.Experimental results,using representative Large Language Models and focusing on Chinese and English,dem-onstrated that these models achieved an accuracy rate of over 80%on cross-lingual cloze tasks,with the highest reaching 90.4%.This performance confirmed the strong multilingual alignment capabilities of Large Language Models.The study not only offered a methodology and data for evaluating multilingual word alignment capabilities but also provided model selection recom-mendations for researchers in multilingual universality and cross-lingual processing tasks.

关 键 词：大语言模型 多语言词对齐能力评测 跨语言选词填空 

分 类 号：TP391[自动化与计算机技术—计算机应用技术] TP18[自动化与计算机技术—计算机科学与技术]