自然语言处理中词语上下文有效范围的定量描述  

Quantitative Analysis of Context Field in Natural Language Processing

作　　者：鲁松[1] 白硕[1] 

机构地区：[1]中国科学院计算技术研究所,北京100080

出　　处：《计算机学报》2001年第7期742-747,共6页Chinese Journal of Computers

基　　金：国家"九七三"高科技发展项目基金 (G19980 3 0 5 10 );国家自然科学基金 (69773 0 0 8);国家"八六三"高技术研究发展计划 (863 -

摘　　要：词语的上下文 (context)是语料库语言学中自然语言知识获取和解决自然语言处理中多种实际应用问题必须依靠的资源和基础 .但上下文“窗口”开多大为宜呢 ?为克服当前仅凭主观经验或通过某一特定应用问题中最终结果正确率界定上下文有效范围的不足 ,我们在引入信息增益方法确定上下文各位置的信息量后 ,构造上下文位置信息量函数 ,最终通过多项式积分确定 85 %信息量的上下文边界 ,即汉语核心词语最近距离 [- 8,+9]和英文[- 16 ,+13]位置之间的上下文范围 .Context is the necessary resource not only for corpus-based linguistic but also for solving the problem in computational linguistics. But what is the size of context window? Because there is no method to define the effective field of context window by quantitative analysis, this paper put forward one computational method to do it with quantitative analysis. First, context position is weighted by information gain; second, the position-weight function is constructed according to weight of context positions, and then the function is integrated to arrive to the information ratio 85% to define the size of windows. The result is [-8, +9] in Chinese and [-16, +13] in English for context window. The result explained quantitatively the value and function of context in natural language processing.

关 键 词：上下文有效范围 信息增益 自然语言处理 词语 计算机 

分 类 号：TP391[自动化与计算机技术—计算机应用技术]