基于多模态推荐指令的大语言模型指令微调  

The Instruction Tuning of Large Language Models with Multi-Modal Recommendation Instruction

作　　者：郝博文 柳溢菲 李立耀 王洁[1] 彭岩[1] HAO Bowen;LIU Yifei;LI Liyao;WANG Jie;PENG Yan(School of Management,Capital Normal University,Beijing 100048,China;School of Mathematical Sciences,Capital Normal University,Beijing 100048,China;Engineering Research Center for ICH Digitalization and Multi-source Information Fusion Fujian Province University,Fujian Polytechnic Normal University,Fuzhou 350300,China)

机构地区：[1]首都师范大学管理学院,北京100048 [2]首都师范大学数学科学学院,北京100048 [3]福建技术师范学院非遗数字化与多源信息融合福建省高校工程研究中心,福州350300

出　　处：《北京邮电大学学报》2024年第4期36-43,共8页Journal of Beijing University of Posts and Telecommunications

基　　金：非遗数字化与多源信息融合福建省高校工程研究中心开放基金(G3-KF2303);国家自然科学基金项目(62172287)。

摘　　要：基于多模态指令的大语言模型指令微调能够有效赋予大模型解决相关多模态任务的能力。为了进一步使大模型能够完成多模态零样本或少样本推荐任务,提出了多模态推荐大语言模型,该模型以大语言模型ChatGLM2-6B为基座,选取包含文本、图片信息的多模态推荐数据集,利用ChatGPT和GPT4构建多模态用户画像和物品属性生成指令,以及零样本和少样本推荐指令,并采用高效参数微调P-tuning v2方式,仅需用一张A100 40GB图形处理器即可微调得到多模态推荐大语言模型,用于完成多模态零样本和少样本推荐任务。实验结果证明,所提模型显著优于现有基线模型。The tuning of large language models based on multimodal instructions has been proven effective in endowing large language models with the capability to address relevant multimodal tasks.To further empower large language models in handling multimodal zero-shot or few-shot recommendation tasks,multi-modal recommendation of large language model is proposed,which is built upon the foundation of ChatGLM2-6B,and is trained on multimodal recommendation dataset that includes both textual and image information.The construction of multimodal user profiles and item attributes is achieved through the utilization of ChatGPT and GPT-4 for generating instructions.Additionally,instructions for zero-shot and few-shot recommendations are formulated.The model undergoes efficient parameter fine-tuning using the P-tuning v2 method,requiring only a single A10040GB graphics processing unit for the fine-tuning process.Experimental results demonstrate that the proposed model significantly outperforms existing baseline models.

关 键 词：多模态推荐指令 大语言模型 指令微调 

分 类 号：TP391[自动化与计算机技术—计算机应用技术]