基于运动估计与时空结合的多帧融合去雨网络  

Motion-estimation Based Space-temporal Feature Aggregation Network for Multi-frames Rain Removal

作　　者：孟祥玉 薛昕惟 李汶霖 王祎[1,2] MENG Xiang-yu;XUE Xin-wei;LI Wen-lin;WANG Yi(DUT-RU International School of Information Science&Engineering of Dalian University of Technology,Dalian,Liaoning 116621,China;Key Laboratory for Ubiquitous Network and Service Software of Liaoning Province,Dalian,Liaoning 116621,China)

机构地区：[1]大连理工大学-立命馆大学国际信息与软件学院,辽宁大连116621 [2]辽宁省泛在网络与服务软件重点实验室,辽宁大连116621

出　　处：《计算机科学》2021年第5期170-176,共7页Computer Science

基　　金：国家自然科学基金(61806036,61976037);中央高校基本科研业务费资助(DUT19TD19)。

摘　　要：降雨天气会导致视觉质量下降,从而影响目标识别和追踪等视觉任务的处理效果。为了减小雨的影响,完成对运动视频背景细节的有效恢复,近年来相关研究者在视频去雨方向提出了很多方法。其中基于卷积神经网络的视频去雨方法使用最为广泛,它们大多采用单帧增强后多帧融合去雨的方式。但由于直接单帧增强使相邻帧之间部分像素的移动无法完成时间维度上的对齐,不能有效实现端到端的训练,因此丢失了大量细节信息,使得最终得到的去雨效果不尽人意。为有效解决上述问题,文中提出了一个基于运动估计与时空结合的多帧融合去雨网络(ME-Derain)。首先通过光流估计算法将相邻帧对齐到当前帧来有效利用时间信息;然后引入基于残差连接的编码器-解码器结构,结合与时间相关的注意力增强机制一起构成多帧融合网络来有效融合多帧信息;最后利用空间相关的多尺度增强模块来进一步增强去雨效果和得到最终的去雨视频。在多个数据集上的大量实验结果表明,所提算法优于现阶段大部分视频去雨算法,能够获得更好的去雨效果。Outdoor videos obtained under rainy weather cause visual quality degradation,which affects the processing effects of visual tasks such as object recognition and tracking.In order to enhance the quality of video and complete the effective recovery of the details in the motion video,many methods have been proposed in video rain removal.At this stage,most of the video rain removal methods based on convolutional neural networks employ single-frame enhancement and multi-frame fusion to remove rain.But the movement of some pixels between adjacent frames with direct enhancement is difficult to be completed in the temporal dimension.And the manner cannot effectively achieve end-to-end training,making the final result still relatively blurry and many detailed information losses.In order to effectively solve the above problems,this paper proposes a multi-frame fusion rain removal network based on the combination of motion estimation and space-temporal feature aggregation,ME-Derain for short.First,the optical flow estimation method is used to establish a reference frame to complete the alignment of adjacent frames,and then an encoder-decoder structure is introduced.The convolutional neural network connected by the residual connection and the time-related attention enhancement mechanism together form a multi-frame fusion network.Finally,the enhancement module related to the spatial sequence is used to obtain the rain removal video.A large number of experiments on different data sets show that the proposed method is better than most common methods at this stage and can obtain better rain removal effect.

关 键 词：视频去雨 光流 卷积神经网络 视频增强 

分 类 号：TP391[自动化与计算机技术—计算机应用技术]