基于LoRA的多主体个性化图像生成方法  

Multi-subject personalized image generation method based on LoRA

作　　者：胡洪峰 周越[1] HU Hongfeng;ZHOU Yue(Shanghai Jiao Tong University,School of Electronic Information and Electrical Engineering,Shanghai 200240,China)

机构地区：[1]上海交通大学电子信息与电气工程学院,上海200240

出　　处：《中国体视学与图像分析》2024年第4期320-332,共13页Chinese Journal of Stereology and Image Analysis

摘　　要：个性化图像生成是指根据用户提供的少量样本来学习某个特定的个性化特征(例如,某个特定人物、物体或风格),并生成具有该对象特征的新图像。虽然针对单一主体的个性化生成方法已经在应用领域取得了巨大的成功,但是针对多个主体的个性化生成场景依然面临着语义混淆和主题消失等诸多挑战。为了可以实现更好的多概念主体个性化生成,本文对流行的单个主体生成的LoRA方法进行了改进,通过对LoRA的训练策略进行优化和增加约束使得LoRA更加适合多主体生成的任务,具有此前不具备的多主体组合能力。本文引入了布局控制的方法,缓解了不同主体在注意力图中的相互干扰,同时在多个数据集上进行了大量的实验。结果表明,本文的方法在多主体个性化生成任务中具有优秀的表现。Personalized image generation refers to the process of learning personalized characteristics,such as those of a particular person,object,or style,from a small number of samples provided by the user,and generating new images that incorporate these characteristics.Methods for generating personalized images of single subject have achieved significant success in various application fields.However,scenarios involving the personalized generation of multiple subjects stil face considerable challenges such as semantic confusion and subject disappearance.To enable better multi-concept subject personalized generation,this work improved the popular single-subject generation method,LoRA,by optimizing its training strategy and incorporating constraints to make it more suitable for multi-subject generation tasks.The enhancement endowed it with multi-subject combination capabilities that it previously lacked.To further alleviate the interference between multiple subjects,we introduced a layout control method to mitigate the mutual interference of different subjects in the attention map.Extensive experiments on multiple datasets demonstrated that our method exhibited excellent performance in multisubject personalized generation tasks.

关 键 词：文生图 扩散模型 个性化 图像生成 

分 类 号：TP391[自动化与计算机技术—计算机应用技术]