A framework for the fusion of visual and tactile modalities for improving robot perception  

作　　者：Wenchang ZHANG Fuchun SUN Hang WU Haolin YANG 

机构地区：[1]The State Key Laboratory of Intelligent Technology and Systems, Tsinghua University, Beijing 100084, China [2]Institution of Medical Equipment, Tianjin 300161, China

出　　处：《Science China(Information Sciences)》2017年第1期141-152,共12页中国科学（信息科学）（英文版）

基　　金：supported by National Natural Science Foundation of China (Grant Nos. 613278050, 61210013, 91420302, 91520201);Academic of Military Medical Science (AMMS) Innovation Foundation (Grant No. 2015CXJJ020)

摘　　要：Robots should ideally perceive objects using human-like multi-modal sensing such as vision, tactile feedback, smell, and hearing. However, the features presentations are different for each modal sensor. Moreover,the extracted feature methods for each modal are not the same. Some modal features such as vision, which presents a spatial property, are static while features such as tactile feedback, which presents temporal pattern,are dynamic. It is difficult to fuse these data at the feature level for robot perception. In this study, we propose a framework for the fusion of visual and tactile modal features, which includes the extraction of features, feature vector normalization and generation based on bag-of-system（BoS）, and coding by robust multi-modal joint sparse representation（RM-JSR） and classification, thereby enabling robot perception to solve the problem of diverse modal data fusion at the feature level. Finally, comparative experiments are carried out to demonstrate the performance of this framework.Robots should ideally perceive objects using human-like multi-modal sensing such as vision, tactile feedback, smell, and hearing. However, the features presentations are different for each modal sensor. Moreover,the extracted feature methods for each modal are not the same. Some modal features such as vision, which presents a spatial property, are static while features such as tactile feedback, which presents temporal pattern,are dynamic. It is difficult to fuse these data at the feature level for robot perception. In this study, we propose a framework for the fusion of visual and tactile modal features, which includes the extraction of features, feature vector normalization and generation based on bag-of-system（BoS）, and coding by robust multi-modal joint sparse representation（RM-JSR） and classification, thereby enabling robot perception to solve the problem of diverse modal data fusion at the feature level. Finally, comparative experiments are carried out to demonstrate the performance of this framework.

关 键 词：multi-modal fusion robot perception vision TACTILE classification 

分 类 号：TP391.41[自动化与计算机技术—计算机应用技术] TP242[自动化与计算机技术—计算机科学与技术]