Temporally consistent video colorization with deep feature propagation and self-regularization learning  

作　　者：Yihao Liu Hengyuan Zhao Kelvin CKChan Xintao Wang Chen Change Loy Yu Qiao Chao Dong 

机构地区：[1]Shenzhen Institute of Advanced Technology,Chinese Academy of Sciences,Shenzhen 518055,China [2]University of Chinese Academy of Sciences,Beijing 100049,China [3]Department of Electrical&Computer Engineering,Nanyang Technological University,50 Nanyang Avenue,Singapore 639798 [4]Applied Research Center,Tencent PCG,Shenzhen,China

出　　处：《Computational Visual Media》2024年第2期375-395,共21页计算可视媒体（英文版）

基　　金：supported by grants from the National Natural Science Foundation of China(61906184);the Joint Lab of CAS–HK,and the Shanghai Committee of Science and Technology,China(20DZ1100800,21DZ1100100).

摘　　要：Video colorization is a challenging and highly ill-posed problem.Although recent years have witnessed remarkable progress in single image colorization,there is relatively less research effort on video colorization,and existing methods always suffer from severe flickering artifacts(temporal inconsistency)or unsatisfactory colorization.We address this problem from a new perspective,by jointly considering colorization and temporal consistency in a unified framework.Specifically,we propose a novel temporally consistent video colorization(TCVC)framework.TCVC effectively propagates frame-level deep features in a bidirectional way to enhance the temporal consistency of colorization.Furthermore,TCVC introduces a self-regularization learning(SRL)scheme to minimize the differences in predictions obtained using different time steps.SRL does not require any ground-truth color videos for training and can further improve temporal consistency.Experiments demonstrate that our method can not only provide visually pleasing colorized video,but also with clearly better temporal consistency than state-of-the-art methods.A video demo is provided at https://www.youtube.com/watch?v=c7dczMs-olE,while code is available at https://github.com/lyh-18/TCVC-Tem porally-Consistent-Video-Colorization.

关 键 词：video colorization temporal consistency feature propagation self-regularization 

分 类 号：TP391.41[自动化与计算机技术—计算机应用技术]