语义推理和联合学习的壁画修复算法  

Mural Inpainting Algorithm Based on Semantic Reasoning and Joint Learning

作　　者：陈永[1,2] 杜婉君 赵梦雪 CHEN Yong;DU Wanjun;ZHAO Mengxue(School of Electronics and Information Engineering,Lanzhou Jiaotong University,Lanzhou 730070,China;Gansu Provincial Engineering Research Center for Artificial Intelligence and Graphics&Image Processing,Lanzhou 730070,China)

机构地区：[1]兰州交通大学电子与信息工程学院,兰州730070 [2]甘肃省人工智能与图形图像处理工程研究中心,兰州730070

出　　处：《北京邮电大学学报》2024年第5期73-80,共8页Journal of Beijing University of Posts and Telecommunications

基　　金：教育部人文社会科学研究青年基金项目(19YJC760012);兰州交通大学基础研究拔尖人才项目(2022JC36);兰州交通大学重点研发项目(ZDYF2304)。

摘　　要：使用深度学习算法修复壁画时,缺乏语义约束,孤立地修复壁画会导致修复结果存在语义不一致,结构纹理紊乱的问题。为了解决这个问题,提出了一种全局语义与局部语义联合学习的壁画修复算法。首先,设计了联合学习分层网络,将壁画分为高层语义和低层语义,实现对不同语义的分层修复;其次,设计了全局联合学习生成模块,通过自回归网络对全局语义进行自回归建模,经语义推理得到壁画的全局修复信息;接着,构建局部联合学习生成模块,提出了上下文聚合块,学习壁画的上下文信息,生成壁画的局部信息;然后,加入联合学习注意力机制,实现全局语义与局部语义的一致性修复,解决了孤立修复导致误差累积和语义不一致的问题;最后,将生成的壁画放入谱归一化判别器中进行判别,得到修复后的壁画。对敦煌壁画的修复实验结果表明,与其他方法相比,用所提方法对壁画进行修复后的壁画具有更好的结构和纹理,所提方法的客观评价指标峰值信噪比和结构相似度均优于其他算法。We propose a mural inpainting algorithm based on semantic inference and dynamic joint learning to address the issues of semantic inconsistency and disordered structural textures caused by the lack of semantic constraints and isolated restoration of texture structure in existing deep learning methods for mural restoration.Firstly,a mural restoration framework based on joint learning is constructed,and a joint hierarchical network is designed to divide the mural into high-level semantics and low-level semantics,enabling hierarchical restoration of different semantics.Secondly,a joint global generation module is designed to model the global semantics of the mural through autoregressive modeling and infer the repaired global semantic information.Next,a joint local generation module is constructed,which introduces a context aggregation block to learn the contextual information of the mural and generate local information for the mural.Then,a joint attention mechanism is introduced to enable collaborative training between the global semantic restoration module and the local restoration module,overcoming the issues of error accumulation and semantic inconsistency caused by isolated restoration.Finally,a joint attention mechanism is introduced to enable collaborative training between the global semantic restoration module and the local restoration module,addressing the issues of error accumulation and semantic inconsistency caused by isolated restoration.Experimental results on real Dunhuang murals show that the proposed method achieves better consistency in restoration compared to other methods,both in terms of peak signal-to-noise ratio and structural similarity.Additionally,the objective evaluation metrics surpass those of the comparison algorithms.

关 键 词：壁画修复 语义推理 联合学习 上下文聚合 联合注意力 

分 类 号：TP391.41[自动化与计算机技术—计算机应用技术]