基于混合时空卷积的轻量级视频超分辨率重建  

Lightweight video super-resolution based on hybrid spatio-temporal convolution

作　　者：夏振平[1,3] 陈豪 张宇宁 程成[1,3] 胡伏原 XIA Zhenping;CHEN Hao;ZHANG Yuning;CHENG Cheng;HU Fuyuan(School of Electronic&Information Engineering,Suzhou University of Science and Technology,Suzhou 215009,China;Display R&D Centre,School of Electronic Science&Engineering,Southeast University,Nanjing 210096,China;Jiangsu Industrial Intelligent and Low-carbon Technology Engineering Center,Suzhou 215009,China;Shi-Cheng Laboratory for Information Display and Visualization,Nanjing 210013,China)

机构地区：[1]苏州科技大学电子与信息工程学院,江苏苏州215009 [2]东南大学电子科学与工程学院显示技术研究中心,江苏南京210096 [3]江苏省工业智能低碳技术工程中心,江苏苏州215009 [4]新型显示与视觉感知石城实验室,江苏南京210013

出　　处：《光学精密工程》2024年第16期2564-2576,共13页Optics and Precision Engineering

基　　金：国家自然科学基金资助项目(No.62002254);江苏省自然科学基金资助项目(No.BK20200988);苏州市科技计划项目(No.SNG-2023002)。

摘　　要：针对三维卷积神经网络在视频超分辨率任务上具有较高的计算复杂度以及提取时空特征有限的问题,本文设计了一种基于混合时空卷积的轻量级视频超分辨率重建网络。首先,提出了一个基于混合时空卷积的模块,实现了网络时空特征提取能力的提升以及计算复杂度的降低;其次,提出了一个基于相似性的选择性特征融合模块,进一步增强了相关特征的提取能力;最后,设计了一种基于注意力机制的运动补偿模块,在一定程度上减轻了错误的特征融合的影响。实验结果表明:所提网络可以在视频超分辨率性能和网络复杂度之间取得很好的平衡,而且在基准数据集SPMCS-11上4倍超分辨率达到8 frame/s。所提网络满足了边缘设备推理运行中快速、准确等要求。Addressing the issue of high computational complexity and limited extraction of spatio-temporal features in 3D convolutional neural networks for video super-resolution tasks,this paper introduced a novel lightweight video super-resolution reconstruction network based on hybrid spatio-temporal convolution.Firstly,a hybrid spatio-temporal convolution-based module was proposed to realize the enhancement of the spatio-temporal feature extraction capability of the network as well as reduction of the computational complexity.Then,a similarity-based selective feature fusion module was proposed to further enhance the extraction capability of relevant features.Lastly,a motion compensation module based on the attention mechanism was designed to mitigate the effects of erroneous feature fusion to a certain extent.The experimental results demonstrate that the proposed network can achieve a favorable balance between video super-resolution performance and network complexity,and the 4-fold super-resolution reaches 8 frames per second on the benchmark dataset SPMCS-11.The proposed network meets the requirements for fast and accurate reasoning operations on edge devices.

关 键 词：视频超分辨率 深度学习 三维卷积神经网络 特征融合 

分 类 号：TP391.41[自动化与计算机技术—计算机应用技术]