单幅图像去雨数据集和深度学习算法的联合评估与展望  

The integrated evaluation and review of single image rain removal based datasets and deep learning methods

作　　者：胡明娣[1] 吴怡 宋尧 杨静冰 张瑞芳 王红[2] 孟德宇[2] Hu Mingdi;Wu Yi;Song Yao;Yang Jingbing;Zhang Ruifang;Wang Hong;Meng Deyu(Xi′an University of Posts and Telecommunications,Xi′an 710121,China;Xi′an Jiaotong University,Xi′an 710049,China)

机构地区：[1]西安邮电大学,西安710121 [2]西安交通大学,西安710049

出　　处：《中国图象图形学报》2022年第5期1359-1391,共33页Journal of Image and Graphics

基　　金：国家自然科学基金项目(62071378);陕西省国际科技合作计划项目(2022KW-04);西安市科技局计划项目(21XJZZ0072)。

摘　　要：雨天会影响室外图像捕捉的质量,进而引起户外视觉任务性能下降。基于深度学习的单幅图像去雨研究因算法性能优越而引起了大家的关注,并且聚焦点集中在数据集的质量、图像去雨方法、单幅图像去雨后续高层任务的研究和性能评价指标等方面。为了方便研究者快速全面了解该领域,本文从上述4个方面综述了基于深度学习的单幅图像去雨的主流文献。依据数据集的构建方式将雨图数据集分为4类:基于背景雨层简单加和、背景雨层复杂融合、生成对抗网络(generative adversarial network,GAN)数据驱动合成的数据集,以及半自动化采集的真实数据集。依据任务场景、采取的学习机制以及网络设计对主流算法分类总结。综述了面向单任务和联合任务的去雨算法,单任务即雨滴、雨纹、雨雾和暴雨的去除;联合任务即雨滴和雨纹、所有噪声去除。综述了学习机制和网络构建方式(比如:卷积神经网络(convolutional neural network,CNN)结构多分支组合,GAN的生成结构,循环和多阶段结构,多尺度结构,编解码结构,基于注意力,基于Transformer)以及数据模型双驱动的构建方式。综述了单幅图像去雨后续高层任务的研究文献和图像去雨算法性能的评价指标。通过合成数据集和真实数据集上的综合实验对比,证实了领域知识隐式引导网络构建可以有效提升算法性能,领域知识显式引导正则化网络的学习有潜力进一步提升算法的泛化性。最后,指出单幅图像去雨工作目前面临的挑战和未来的研究方向。The visual quality of captured images in rainy weather conditions is constrained to the outdoor vision degradation.It is essential to design relevant rain removal algorithms.Due to the lack of temporal information,single image rain removal is challenging compared to video-based rain removal.The target of single image rain removal analysis is to restore the rain-removed background image from the corresponding rain-affected image.Current deep learning based vision tasks construct diverse data-driven frameworks like single image rain removal task via multiple network modules.Current research tasks are focusing on the quality of datasets,the design of single image deraining algorithms,the subsequent high-level vision tasks,and the design of performance evaluation metrics.Specifically,the quality of rain datasets largely affects the performance of deep learning based single image deraining methods,since the generalization ability of deep single image rain removal is highly related to the domain gap between synthesized training dataset and real testing dataset.Besides,rain removal plays an important preprocessing role in outdoor visual tasks because its result would affect the performance of the subsequent visual task.Additionally,the design of image quality assessment(IQA)metrics is quite important for the fair quantitative analysis of human perception of image quality in general image restoration tasks.We conducted critical literature review for deep learning based single image rain removal from the four aspects as mentioned below:1)dataset generation in rain weather conditions;2)representative deep neural network based single image rain removal algorithms;3)the research of the downstream high-level task in rainy days and 4)performance metrics for evaluating single image rain removal algorithms.Specifically,in terms of the generation manners,the current rain image datasets are roughly divided into four categories as following:1)synthesizing rain streaks based on photo-realistic rendering technique and then adding the

关 键 词：单幅图像去雨 深度神经网络 雨图数据集 雨图合成 模型数据双驱动 后续高层任务 性能评价指标 

分 类 号：TP391[自动化与计算机技术—计算机应用技术]