Semantics-aware transformer for 3D reconstruction from binocular images  

作　　者：JIA Xin YANG Shourui GUAN Diyi 

机构地区：[1]the engineering research center of learning-based intelligent system and the key laboratoryof computer vision and system of ministry of education,tianjin university of technology,tianjin 300384,china [2]zhejiang university of technology,hangzhou 310014,china

出　　处：《Optoelectronics Letters》2022年第5期293-299,共7页光电子快报（英文版）

基　　金：supported by the National Key R&D Program of China (No.2018YFB1305200);the National Natural Science Foundation of China (Nos.61906134, 62020106004, 92048301, and 61925201)

摘　　要：Existing multi-view three-dimensional(3 D) reconstruction methods can only capture single type of feature from input view, failing to obtain fine-grained semantics for reconstructing the complex shapes. They rarely explore the semantic association between input views, leading to a rough 3 D shape. To address these challenges, we propose a semantics-aware transformer(SATF) for 3 D reconstruction. It is composed of two parallel view transformer encoders and a point cloud transformer decoder, and takes two red, green and blue(RGB) images as input and outputs a dense point cloud with richer details. Each view transformer encoder can learn a multi-level feature, facilitating characterizing fine-grained semantics from input view. The point cloud transformer decoder explores a semantically-associated feature by aligning the semantics of two input views, which describes the semantic association between views. Furthermore, it can generate a sparse point cloud using the semantically-associated feature. At last, the decoder enriches the sparse point cloud for producing a dense point cloud with richer details. Extensive experiments on the Shape Net dataset show that our SATF outperforms the state-of-the-art methods.

关 键 词：DETAILS SEMANTIC DECODER 

分 类 号：TP391.41[自动化与计算机技术—计算机应用技术]