事件融合与空间注意力和时间记忆力的视频去雨网络  

Event-fusion-based Spatial Attentive and Temporal Memorable Network for Video Deraining

作　　者：孙上荃 任文琦 操晓春 SUN Shang-Quan;REN Wen-Qi;CAO Xiao-Chun(Institute of Information Engineering,Chinese Academy of Sciences,Beijing 100085,China;School of Cyber Security,University of Chinese Academy of Sciences,Beijing 100049,China;School of Cyber Science and Technology,Sun Yat-sen University,Shenzhen 518107,China)

机构地区：[1]中国科学院信息工程研究所,北京100085 [2]中国科学院大学网络空间安全学院,北京100049 [3]中山大学网络空间安全学院,广东深圳518107

出　　处：《软件学报》2024年第5期2220-2234,共15页Journal of Software

基　　金：国家自然科学基金(62172409);深圳市科技计划(JCYJ20220530145209022)。

摘　　要：近年来数码视频拍摄设备不断升级,其感光元件宽容度、快门速率的提升虽然极大程度地丰富了可拍摄景物的多样性,雨痕这类由于雨滴高速穿过景深范围的退化元素也更容易被记录到,作为前景的稠密雨痕阻挡了背景景物的有效信息,从而影响图像的有效采集.由此视频图像去雨成为一个亟待解决的问题,以往的视频去雨方法集中在利用常规图像自身的信息,但是由于常规相机的感光元件物理极限、快门机制约束等原因,许多光学信息在采集时丢失,影响后续的视频去雨效果.由此,利用事件数据与常规视频信息的互补性,借助事件信息的高动态范围、时间分辨率高等优势,提出基于事件数据融合与空间注意力和时间记忆力的视频去雨网络,利用三维对齐将稀疏事件流转化为与图像大小匹配的表达形式,叠加输入至集合了空间注意力机制的事件-图像融合处理模块,有效提取图像的空间信息,并在连续帧处理时使用跨帧记忆力模块将先前帧特征利用,最后经过三维卷积与两个损失函数的约束.在开源视频去雨数据集上验证所提方法的有效性,同时达到了实时视频处理的标准.In recent years,digital video shooting equipment has been continuously upgraded.Although the improvement of the latitude of its image sensor and shutter rate has greatly enriched the diversity of the scene that can be photographed,the degraded factors such as rain streaks caused by raindrops passing through the field of view at high speed are also easier to be recorded.The dense rain streaks in the foreground block the effective information of the background scene,thus affecting the effective acquisition of images.Therefore,video image deraining becomes an urgent problem to be solved.The previous video deraining methods focus on using the information of conventional images themselves.However,due to the physical limit of the image sensors of conventional cameras,the constraints of the shutter mechanism,etc.,much optical information is lost during video acquisition,which affects the subsequent video deraining effect.Therefore,taking advantage of the complementarity of event data and conventional video information,as well as the high dynamic range and high temporal resolution of event information,this study proposes a video deraining network based on event data fusion,spatial attention,and temporal memory,which uses three-dimensional alignment to convert the sparse event stream into an expression form that matches the size of the image and superimposes the input to the event-image fusion module that integrates the spatial attention mechanism,so as to effectively extract the spatial information of the image.In addition,in continuous frame processing,the inter-frame memory module is used to utilize the previous frame features,which are finally constrained by the three-dimensional convolution and two loss functions.The video deraining method is effective on the publicly available dataset and meets the standard of real-time video processing.

关 键 词：视频去雨 事件数据 多模态融合 空间注意力 时间记忆力 

分 类 号：TP391[自动化与计算机技术—计算机应用技术]