面向语文辞书编纂的神经网络语料库检索研究  

Neural Corpus Retrieval for Language Dictionary Compilation

作　　者：胡钦谙 Hu Qinan

机构地区：[1]中国社会科学院语言研究所/辞书编纂研究中心,北京100732

出　　处：《辞书研究》2023年第1期36-45,I0002,共11页Lexicographical Studies

基　　金：国家语委“十三五”科研规划2020年度一般(委托)项目“辅助语文辞书编纂的人工智能关键技术研究”(项目编号WT135-69)的研究成果之一。

摘　　要：语料库检索已成为语文辞书编纂过程中不可或缺的一个步骤。现有的语料库检索系统通常先返回句子列表,然后由人工进行筛选。然而,随着语料库规模不断扩大,海量检索结果与用户有限的语料消化能力之间的矛盾已成为辞书编纂过程中的痛点之一。文章注意到,辞书编纂人员在检索语料库时,实际上更为关注的是检索词出现的语境(或称上下文)。语境具有相对稳定的语言学规律,语料库中往往存在着大量符合同一语言规律的语境。这种同质的语境对辞书编纂提供的价值相对有限,单纯依靠增加语料数量对辞书编纂的贡献已呈现出边际效益递减的现象。因此,文章尝试以海量检索结果中的同质语境为突破口,通过人工智能中的注意力机制、上下文相关词向量以及预训练语言模型等技术,以可视化及可交互的形式为辞书编纂人员展现检索结果分布的概貌,批量处理同质的语境,以提升辞书编纂的效率。Corpora have become indispensable to language dictionary compilation.A corpus retrieval system usually returns a list of sentences that demand further manual cherry-picking.However,while the scale of the corpus has grown dramatically,the contradictions between massive search results and a limited digestive capability of human beings have become one of the pains in dictionary compilation.As is noticed,it is the context in which keywords are used that dictionary editors are most interested in.There typically exist a large number of homogenous contexts following the same constant linguistic rules in corpora.These contexts are of limited values to lexicography.Following the law of diminishing marginal utility,new additions to corpora have contributed less and less to dictionary compilation.This study takes homogeneous contexts as a pathway to understanding massive search results.Making use of the technologies of artificial intelligence,including attentions,contextual word embeddings,and pre-trained language models,it helps dictionary editors to grasp an overview of search results visually and interactively,handle homogeneous contexts in batch,and eventually improve the efficiency of dictionary compilation.

关 键 词：语料库检索 语境 语义 上下文相关词向量 注意力机制 

分 类 号：TP183[自动化与计算机技术—控制理论与控制工程] TP391.1[自动化与计算机技术—控制科学与工程] H16[语言文字—汉语]