多模态人机交互综述  

A survey on multi-modal human-computer interaction

作　　者：陶建华[1] 巫英才 喻纯[3] 翁冬冬[4] 李冠君 韩腾 王运涛[3] 刘斌[1] Tao Jianhua;Wu Yingcai;Yu Chun;Weng Dongdong;Li Guanjun;Han Teng;Wang Yuntao;Liu Bin(Institute of Automation,Chinese Academy of Sciences,Beijing 100190,China;Zhejiang University,Hangzhou 310058,China;Tsinghua University,Beijing 100084,China;Beijing Institute of Technology,Beijing 100081,China;Institute of Software,Chinese Academy of Sciences,Beijing 100190,China)

机构地区：[1]中国科学院自动化研究所,北京100190 [2]浙江大学,杭州310058 [3]清华大学,北京100084 [4]北京理工大学,北京100081 [5]中国科学院软件研究所,北京100190

出　　处：《中国图象图形学报》2022年第6期1956-1987,共32页Journal of Image and Graphics

摘　　要：多模态人机交互旨在利用语音、图像、文本、眼动和触觉等多模态信息进行人与计算机之间的信息交换。在生理心理评估、办公教育、军事仿真和医疗康复等领域具有十分广阔的应用前景。本文系统地综述了多模态人机交互的发展现状和新兴方向,深入梳理了大数据可视化交互、基于声场感知的交互、混合现实实物交互、可穿戴交互和人机对话交互的研究进展以及国内外研究进展比较。本文认为拓展新的交互方式、设计高效的各模态交互组合、构建小型化交互设备、跨设备分布式交互、提升开放环境下交互算法的鲁棒性等是多模态人机交互的未来研究趋势。Benefiting from the development of the Internet of things,human-computer interaction devices have been widely used in people’s daily life.Human-computer interaction is no longer limited to the input and output modes of a single sensory channel(vision,touch,hearing,smell and taste).Multi-modal human-computer interaction aims to exchange information between human and computer by using multi-modal information such as speech,image,text,eye movement and touch.Multi-modal human-computer interaction includes multi-modal information input from human to computer and multimodal information presentation from computer to human and it is a comprehensive discipline closely related to cognitive psychology,ergonomics,multimedia technology and virtual reality technology.At present,multi-modal human-computer interaction and various kinds of academic and technology in the field of image and graphics are more and more closely combined.In the era of big data and artificial intelligence,multi-modal human-computer interaction technology,as the technical carrier of human-machine-thing,is closely related to the development of image and graphics,artificial intelligence,emotional computing,physiological and psychological assessment,Internet big data,office education,medical rehabilitation and other fields.The research on multi-modal human-computer interaction first appeared in the 1990 s,and a number of works proposed an interactive method combining speech and gesture.In recent years,the emergence of immersive visualization provides a new multi-modal interactive interface for human-computer interaction:an immersive environment that integrates visual,auditory,tactile and other sensory channels.Visualization is an important scientific technology for data analysis and exploration.It converts abstract data into graphical representations and facilitates analytical reasoning through interactive interfaces.In today’s data explosion,visualization transforms complex big data into easy-to-understand content,improving people’s ability to unders

关 键 词：多模态人机交互 大数据可视化交互 声场感知交互 实物交互 可穿戴交互 人机对话交互 

分 类 号：TP391.4[自动化与计算机技术—计算机应用技术]