A flexible technique to select objects via convolutional neural network in VR space  

作　　者：Huiyu LI Linwei FAN 

机构地区：[1]School of Computer Science and Technology,Shandong University,Jinan 250101,China [2]Shandong Province Key Lab of Digital Media Technology,Shandong University of Finance and Economics,Jinan 250061,China [3]Shandong Co-Innovation Center of Future Intelligent Computing,Shandong Technology and Business University,Yantai 264005,China

出　　处：《Science China(Information Sciences)》2020年第1期53-72,共20页中国科学（信息科学）（英文版）

摘　　要：Most studies on the selection techniques of projection-based VR systems are dependent on users wearing complex or expensive input devices, however there are lack of more convenient selection techniques.In this paper, we propose a flexible 3 D selection technique in a large display projection-based virtual environment. Herein, we present a body tracking method using convolutional neural network(CNN) to estimate3 D skeletons of multi-users, and propose a region-based selection method to effectively select virtual objects using only the tracked fingertips of multi-users. Additionally, a multi-user merge method is introduced to enable users’ actions and perception to realign when multiple users observe a single stereoscopic display.By comparing with state-of-the-art CNN-based pose estimation methods, the proposed CNN-based body tracking method enables considerable estimation accuracy with the guarantee of real-time performance. In addition, we evaluate our selection technique against three prevalent selection techniques and test the performance of our selection technique in a multi-user scenario. The results show that our selection technique significantly increases the efficiency and effectiveness, and is of comparable stability to support multi-user interaction.Most studies on the selection techniques of projection-based VR systems are dependent on users wearing complex or expensive input devices, however there are lack of more convenient selection techniques.In this paper, we propose a flexible 3 D selection technique in a large display projection-based virtual environment. Herein, we present a body tracking method using convolutional neural network(CNN) to estimate3 D skeletons of multi-users, and propose a region-based selection method to effectively select virtual objects using only the tracked fingertips of multi-users. Additionally, a multi-user merge method is introduced to enable users’ actions and perception to realign when multiple users observe a single stereoscopic display.By comparing with state-of-the-art CNN-based pose estimation methods, the proposed CNN-based body tracking method enables considerable estimation accuracy with the guarantee of real-time performance. In addition, we evaluate our selection technique against three prevalent selection techniques and test the performance of our selection technique in a multi-user scenario. The results show that our selection technique significantly increases the efficiency and effectiveness, and is of comparable stability to support multi-user interaction.

关 键 词：convolutional neural network interaction techniques pose estimation virtual reality 3D selection 

分 类 号：TP391.9[自动化与计算机技术—计算机应用技术] TP183[自动化与计算机技术—计算机科学与技术]