基于扩散模型微调的局部定制图像编辑算法  

Local customized image editing algorithm based on diffusion model fine tuning

作　　者：杜佳俊 兰红[1] 王超凡 Du Jiajun;Lan Hong;Wang Chaofan(College of Information Engineering,Jiangxi University of Science&Technology,Ganzhou Jiangxi 341000,China;Guangdong Provincial Key Laboratory of Diabetology,Dept.of Endocrinology&Metabolism,The Third Affiliated Hospital of Sun Yat-sen University,Guangzhou 510630,China)

机构地区：[1]江西理工大学信息工程学院,江西赣州341000 [2]中山大学附属第三医院内分泌与代谢病学科广东省糖尿病防治重点实验室,广州510630

出　　处：《计算机应用研究》2025年第2期623-629,共7页Application Research of Computers

基　　金：广州市科技计划资助项目(2023A04J1087)。

摘　　要：针对现有基于扩散模型的图像编辑方法存在无法灵活控制图像编辑区域以及生成个性化内容等问题,提出一种基于扩散模型微调的局部定制图像编辑算法。该方法借助稳定扩散模型作为基础框架,首先从给定的一组图像和词嵌入中学习概念嵌入,并且为了提高模型的训练效率,解决由少量数据训练而产生的过拟合问题,在微调过程中通过分析训练过程中各层参数变化的程度降低训练参数数量;然后在联合分割模型中通过局部选择步骤得到掩码特征,进一步精确识别编辑区域边界,从而保护了非编辑区域内容;最后将参考图像、掩码特征和与定制概念绑定相关的条件文本描述共同输入微调模型中,使其在编辑区域精确生成定制内容,增加了在编辑区域生成用户定制内容的灵活性。在DreamBench数据集上的实验结果显示,相较于其他先进方法,该方法在CLIP-T、MS-SSIM评价指标上分别提高了12.2%、13.9%,表明该方法在文本对齐和结构一致性等方面均优于其他的主流方法,为用户提供了更加准确的个性化概念图像编辑方法。In response to the limitations of existing image editing methods based on diffusion models,such as inflexible control over editing regions and the generation of personalized content,this paper proposed a locally customized image editing algorithm based on fine-tuning of diffusion models.Leveraging a stable diffusion model as the foundational framework,the method initially learnt concept embeddings from a given set of images and word embeddings.To enhance training efficiency and mitigate overfitting caused by limited data,the method reduced the number of training parameters during fine-tuning by analyzing the degree of parameter changes across layers during training.Subsequently,in the joint segmentation model for local selection,it obtained mask features to precisely identify the boundaries of the editing area,thereby preserving the content of non-editing areas.Finally,it jointly input the reference image,mask features,and condition text descriptions associated with customized concepts into the fine-tuning model,enabling precise generation of customized content in the editing area and enhancing flexibility in generating user-customized content in the editing area.Experimental results on the DreamBench dataset demonstrate that compared to other state-of-the-art methods,the method achieves the best experimental results,with improvements of 12.2%and 13.9%in CLIP-T and MS-SSIM,respectively.This indicates that the method outperforms mainstream methods in text alignment and structural consistency,providing users with a more accurate personalized concept image editing approach.

关 键 词：图像编辑 扩散模型 模型微调 局部定制 

分 类 号：TP391.41[自动化与计算机技术—计算机应用技术]