大语言模型辅助下的增强现实装配方法  

Augmented Reality Assembly Method Assisted by Large Language Models

作　　者：鲍劲松[1] 李建军 袁轶 吕超凡 王森[3] BAO Jinsong;LI Jianjun;YUAN Yi;LÜChaofan;WANG Sen(Donghua University,Shanghai 201620,China;Shanghai Waigaoqiao Shipbuilding Co.,Ltd.,Shanghai 200137,China;Shanghai Baosight Software Limited Company,Shanghai 201900,China)

机构地区：[1]东华大学,上海201620 [2]上海外高桥造船有限公司,上海200137 [3]上海宝信软件股份有限公司,上海201900

出　　处：《航空制造技术》2024年第16期107-116,共10页Aeronautical Manufacturing Technology

基　　金：国家重点研发计划(2019YFB1706300)。

摘　　要：基于增强现实的装配引导系统将数字信息叠加到物理场景中,有效指导了复杂装配作业任务。然而装配环境中人与物理世界的隔阂仍然巨大,待融合到物理世界的信息需事先准备好,并且需要人工在装配过程中来触发。研究实时且无处不在的提示,成为当前增强现实环境下的复杂装配研究热点,本文提出了一种基于大语言模型(LLMs)辅助的增强现实装配方法,其核心是将LLMs作为装配过程中的另外一个大脑,提供无处不在的装配引导和工艺信息提示支持。首先,建立了LLMs辅助的增强现实装配方法体系,分析了体系的要素及关联关系。其次,面向LLMs环境,构建了匹配的工艺信息模型。随后,给出了基于LLMs的辅助引导装配方法和流程。最后,结合某线缆装配专业知识,研发了一个专业问答系统,实现了LLMs智能辅助引导,使装配合格率提升了15%,并通过多个案例验证了该方法的有效性。Augmented reality-based assembly guidance systems superimpose digital information onto physical scenes to effectively guide complex assembly tasks.However,the gap between humans and the physical world in the assembly environment is huge,and the information to be fused into the physical world needs to be prepared in advance and triggered manually during the assembly process.The study of real-time and ubiquitous prompts is a hot research topic for complex assembly in augmented reality environment.In this paper,we propose an augmented reality assembly method based on the assistance of large language models(LLMs),the core of which is to use LLMs as another brain in the assembly process,providing ubiquitous assembly guidance and prompts support for technique information.Firstly,a system of LLMs-assisted augmented reality assembly method is established,and the elements and interrelationships of the system are analysed.Secondly,a matching process information model is constructed for LLMs environment.Then,the assisted guidance decision-making method and process based on LLMs are given.Finally,combined with a cable assembly expertise,a professional question answering system is developed to realize the smart assisted guidance of LLMs.Results show that the assembly pass rate increased by 15%and the effectiveness of the method is verifed through several cases.

关 键 词：增强现实 大语言模型(LLMs) 装配 问答系统 知识图谱 

分 类 号：TP391.9[自动化与计算机技术—计算机应用技术] TG95[自动化与计算机技术—计算机科学与技术]