对话系统评价方法综述  

Survey of evaluation methods for dialogue systems

作　　者：张伟男[1] 张杨子[1] 刘挺[1] 

机构地区：[1]哈尔滨工业大学社会计算与信息检索研究中心,哈尔滨150001

出　　处：《中国科学：信息科学》2017年第8期953-966,共14页Scientia Sinica(Informationis)

基　　金：国家重点基础研究发展计划(批准号:2014CB340503);国家自然科学基金(批准号:61502120;61472105)资助项目

摘　　要：本文介绍了对话系统的发展历史以及随着对话系统发展而衍生出的多种对话系统评价方法,从任务型对话系统与开放域对话系统两个方向进行了调研和总结,分析了不同评价方法的利弊,每种评价方法的侧重点和不同方向上最新的研究进展.在任务型对话系统方面,介绍了Steve Young等人的近期研究成果,总结了几种被广泛使用的评价思路.在开放域对话系统方面,从客观指标评价和模拟人工评分两个角度探索了开放域聊天系统的评价方法,对于不同的指标和不同的研究思路做了分析及介绍.最后,本文通过总结及分析对话系统的经典评价方法和当前最新的基于神经网络模型的对话评价方法,对对话系统评价方法的发展趋势进行了展望.This paper introduces the history of dialogue systems and their evaluation methods. The evaluation methods are categorized as either task-oriented dialogue systems or open domain dialogue systems. This paper investigates and summarizes the different methods of evaluating dialogue systems, analyzes the pros and cons of the different methods, discusses the emphasis of each method, and presents the progress of recent research for the two categories. For task-oriented dialogue systems, this paper introduces the recent research results of Steve Young. In addition, this paper sums up several widely used evaluation approaches. The evaluation methods for open domain chatting systems are explored from two angles： objective index evaluation and simulated artificial scoring. The various indices and different research ideas are analyzed and introduced as well. Finally, through summarizing and analyzing classical evaluation methods of dialogue systems as well as the newer evaluation methods based on neural network models, this study aims to predict developmental trends in evaluation methods for dialogue systems.

关 键 词：对话系统评价方法 开放域对话系统 任务型对话系统 自然语言处理 人工智能 

分 类 号：TP18[自动化与计算机技术—控制理论与控制工程]