Fusion of color and hallucinated depth features for enhanced multimodal deep learning-based damage segmentation  

作　　者：Tarutal Ghosh Mondal Mohammad Reza Jahanshahi 

机构地区：[1]Civil,Architectural and Environmental Engineering,Missouri University of Science and Technology,Rolla,MO,USA [2]Lyles School of Civil Engineering,Purdue University,West Lafayette,IN,USA [3]School of Electrical and Computer Engineering,Purdue University,West Lafayette,IN,USA

出　　处：《Earthquake Engineering and Engineering Vibration》2023年第1期55-68,共14页地震工程与工程振动（英文刊）

基　　金：supported in part by a fund from Bentley Systems,Inc.

摘　　要：Recent advances in computer vision and deep learning have shown that the fusion of depth information can significantly enhance the performance of RGB-based damage detection and segmentation models.However,alongside the advantages,depth-sensing also presents many practical challenges.For instance,the depth sensors impose an additional payload burden on the robotic inspection platforms limiting the operation time and increasing the inspection cost.Additionally,some lidar-based depth sensors have poor outdoor performance due to sunlight contamination during the daytime.In this context,this study investigates the feasibility of abolishing depth-sensing at test time without compromising the segmentation performance.An autonomous damage segmentation framework is developed,based on recent advancements in vision-based multi-modal sensing such as modality hallucination(MH)and monocular depth estimation(MDE),which require depth data only during the model training.At the time of deployment,depth data becomes expendable as it can be simulated from the corresponding RGB frames.This makes it possible to reap the benefits of depth fusion without any depth perception per se.This study explored two different depth encoding techniques and three different fusion strategies in addition to a baseline RGB-based model.The proposed approach is validated on computer-generated RGB-D data of reinforced concrete buildings subjected to seismic damage.It was observed that the surrogate techniques can increase the segmentation IoU by up to 20.1%with a negligible increase in the computation cost.Overall,this study is believed to make a positive contribution to enhancing the resilience of critical civil infrastructure.

关 键 词：multimodal data fusion depth sensing vision-based inspection UAV-assisted inspection damage segmentation post-disaster reconnaissance modality hallucination monocular depth estimation 

分 类 号：P315.9[天文地球—地震学]