Multimodal interaction design and application in augmented reality for chemical experiment  

作　　者：Mengting XIAO Zhiquan FENG Xiaohui YANG Tao XU Qingbei GUO 

机构地区：[1]Department of Information Science and Engineering,University of Jinan,Jinan 250022,China [2]Shandong Provincial Key Laboratory of Network Based Intelligent Computing,Jinan 250022,China

出　　处：《Virtual Reality & Intelligent Hardware》2020年第4期291-304,共14页虚拟现实与智能硬件（中英文）

基　　金：the National Key R&D Program of China(2018YFB1004901);the Independent Innovation Team Project of Jinan City(2019GXRC013).

摘　　要：Background Augmented reality classrooms have become an interesting research topic in the field of education,but there are some limitations.Firstly,most researchers use cards to operate experiments,and a large number of cards cause difficulty and inconvenience for users.Secondly,most users conduct experiments only in the visual modal,and such single-modal interaction greatly reduces the users'real sense of interaction.In order to solve these problems,we propose the Multimodal Interaction Algorithm based on Augmented Reality(ARGEV),which is based on visual and tactile feedback in Augmented Reality.In addition,we design a Virtual and Real Fusion Interactive Tool Suite(VRFITS)with gesture recognition and intelligent equipment.Methods The ARGVE method fuses gesture,intelligent equipment,and virtual models.We use a gesture recognition model trained by a convolutional neural network to recognize the gestures in AR,and to trigger a vibration feedback after a recognizing a five finger grasp gesture.We establish a coordinate mapping relationship between real hands and the virtual model to achieve the fusion of gestures and the virtual model.Results The average accuracy rate of gesture recognition was 99.04%.We verify and apply VRFITS in the Augmented Reality Chemistry Lab(ARCL),and the overall operation load of ARCL is thus reduced by 29.42%,in comparison to traditional simulation virtual experiments.Conclusions We achieve real-time fusion of the gesture,virtual model,and intelligent equipment in ARCL.Compared with the NOBOOK virtual simulation experiment,ARCL improves the users'real sense of operation and interaction efficiency.

关 键 词：Augmented reality Gesture recognition Intelligent equipment Multimodal Interaction Augmented Reality Chemistry Lab 

分 类 号：TP3[自动化与计算机技术—计算机科学与技术]