跨模态语言大模型:进展及展望  

Cross-modal Large Language Models:Progress and Prospects

作　　者：陈露[1,2] 张思拓 俞凯[1,2] Lu Chen;Situo Zhang;Kai Yu(X-LANCE Lab,Department of Computer Science and Engineering,Shanghai Jiao Tong University,Shanghai 200240;MoE Key Lab of Artificial Intelligence,AI Institute,Shanghai Jiao Tong University,Shanghai 200240)

机构地区：[1]上海交通大学计算机科学与工程系跨媒体语言智能实验室,上海200240 [2]上海交通大学人工智能教育部重点实验室,上海200240

出　　处：《中国科学基金》2023年第5期776-785,共10页Bulletin of National Natural Science Foundation of China

基　　金：国家自然科学基金项目(62120106006,62106142);上海市市级科技重大专项(2021SHZDZX0102)的资助。

摘　　要：以ChatGPT为代表的对话式语言大模型通过使用超大规模模型参数和海量训练数据,涌现出很强的上下文学习能力和思维链推理能力,在各种自然语言处理任务上取得了显著的进步,被视为颠覆性通用人工智能技术。在纯文本语言大模型突破的基础上,近期显现的重要技术发展趋势是向能够理解和生成语音、图像、图形等其他模态数据的跨模态语言大模型的转变。随着大模型技术的快速发展,跨模态语言大模型逐步拥有了较强的多模态感知以及初步的跨模态认知能力。本文将从多模态感知大模型、跨模态认知大模型、以及分布式智能体系统三种范式综述跨模态语言大模型技术体系的演进过程,并总结相关的评测基准,最后讨论跨模态语言大模型面临的技术挑战及潜在重要研究方向。Conversational large language models(LLMs),such as ChatGPT,have achieved remarkable advancements in in-context learning and reasoning abilities by utilizing massive training data and large-scale model parameters.Building upon the breakthroughs in text-based language models,there has recently been a significant technological trend towards understanding and generating other modalities,such as speech,images,and graphics.This trend has led to the transition into cross-modal LLMs.With the rapid development of large models,cross-modal LLMs have gradually acquired strong multimodal perception and initial cross-modal cognitive abilities.This article first provides a comprehensive overview of the evolution of cross-modal LLM technology from three perspectives:multimodal large perception models,cross-modal large cognitive models,and distributed agent systems,then summarizes the relevant evaluation benchmarks.Additionally,the article discusses the technical challenges and potential research directions that cross-modal LLMs are currently facing.

关 键 词：语言大模型 多模态感知 跨模态认知 分布式智能体 

分 类 号：TP18[自动化与计算机技术—控制理论与控制工程] TP391.1[自动化与计算机技术—控制科学与工程]