利用图像平滑结构信息指导图像修复  

Using image smoothing structure information to guide image inpainting

作　　者：张家骏 廉敬[1,2] 刘冀钊 董子龙[1] 张怀堃 ZHANG Jiajun;LIAN Jing;LIU Jizhao;DONG Zilong;ZHANG Huaikun(School of Electronic and Information Engineering,Lanzhou Jiaotong University,Lanzhou 730000,China;School of Information Science and Engineering,Lanzhou University,Lanzhou 730000,China)

机构地区：[1]兰州交通大学电子与信息工程学院,甘肃兰州730000 [2]兰州大学信息科学与工程学院,甘肃兰州730000

出　　处：《光学精密工程》2024年第4期549-564,共16页Optics and Precision Engineering

基　　金：国家自然科学基金资助项目(No.62061023,No.82260364,No.61941109);甘肃省杰出青年学者(No.21JR7RA345);甘肃省自然科学基金资助项目(No.22JRJ5RA166,No.21JR1RA024,No.21JR1RA252);甘肃省科技计划资助项目(No.20JR10RA273)。

摘　　要：利用图像结构特征进行图像修复,是近年来在深度学习技术广泛应用背景下出现的新方法。应用该方法可以在缺失区域内生成合理的内容,但图像修复结果过于依赖图像结构的提取内容,且在实际训练中会出现错误的持续传播和累积,一旦图像结构存在噪声或失真会直接影响到图像的生成质量。该方法处在探索应用阶段,尚存在网络训练难度大、鲁棒性较差、生成图像上下文语义不一致等问题。为此,本文提出了一种图像平滑结构指导修复的并行网络结构。图像平滑结构的生成内容不直接作为下一级网络的输入,只为网络的解码层提供指导信息。同时,为了更好地匹配和均衡结构与图像之间的特征关系,本文结合transformer提出了一种多尺度特征指导模块。该模块利用transformer联系全局特征的强大建模能力,对结构和图像纹理之间的特征进行匹配和均衡。实验结果表明,本文方法在三个常用的数据集上能够有效地恢复图像缺损内容,并且可以作为图像编辑工具实现目标移除。Using image structure features for image inpainting is a new method that has emerged in recent years with the widespread application of deep learning techniques.This method can generate plausible con⁃tent within missing areas,but the restoration results heavily rely on the extracted content of image struc⁃tures.In practical training,errors can propagate and accumulate,directly impacting the quality of the gen⁃erated image when there is noise or distortion in the image structure.This method is still in the exploratory phase and faces challenges such as difficulty in network training,poor robustness,and inconsistent seman⁃tic context in generated images.To address these issues,this paper proposed a parallel network structure for image inpainting guided by smooth image structures. The generated content of the smooth image struc⁃ ture was not directly used as input for the next-level network but served as guidance information for the de⁃ coding layer of network. Additionally, to better match and balance the feature relationship between the structure and the image, this paper combined transformer and introduces a multi-scale feature guidence module. This module utilized the powerful modeling capability of transformers to establish connections be⁃ tween global features, matching and balancing features between structure and image textures. Experimen⁃ tal results demonstrate that the proposed method effectively restores missing content in images on three commonly used datasets and can be used as an image editing tool for object removal.

关 键 词：图像修复 深度学习 平滑结构 TRANSFORMER 

分 类 号：TP391.41[自动化与计算机技术—计算机应用技术]