面向金融领域大语言模型的提示注入攻击防御机制研究及应用  

Research and application of defense mechanism for prompt injection attack of large language model in financial industry

作　　者：牟大恩 卫志华[1] 孙铭隆 宋娜 倪琳 MOU Daen;WEI Zhihua;SUN Minglong;SONG Na;NI Lin(School of Electronic and Information Engineering,Tongji University,Shanghai 200092,China;Shanghai Securities Co.LTD,Shanghai 200002,China)

机构地区：[1]同济大学电子与信息工程学院,上海200092 [2]上海证券有限责任公司,上海200002

出　　处：《网络与信息安全学报》2024年第5期119-133,共15页Chinese Journal of Network and Information Security

基　　金：国家自然科学基金(62376199)。

摘　　要：大语言模型在金融领域具有广阔的应用前景,无论资产管理端还是财富管理端,大语言模型都将会发挥重要的作用。随着ChatGPT和GPT-4等大型语言模型的快速发展和广泛应用,大语言模型的安全问题也愈发受到人们的关注。金融行业是一个严监管和强监管行业,对安全性有着更高的要求。围绕金融行业大语言模型提示注入攻击问题及安全防御机制进行深入研究,构建了包含8类输入侧提示注入攻击形式和5类大语言模型输出侧安全场景的金融领域大模型风险分类体系,通过人机对抗方式构建了金融领域的大模型提示注入攻击数据集FIN-CSAPrompts,提出了一套端到端的针对大语言模型提示注入攻击的安全防御框架,选取主流的中文大模型进行了实验验证。结果表明,在金融领域应用该提示词防御框架,中文大模型的防御性能显著提升,有效减少了不当内容的生成,并增强了模型对提示注入攻击的抵抗力。为进一步研究中文大模型在金融领域的安全性提供了数据集、衡量指标及解决方案的参考与借鉴。The large language models had a broad application prospect in the financial sector,and they were expected to play an important role in both asset management and wealth management.With the rapid development and wide application of large language models such as ChatGPT and GPT-4,attention to the security of large language models increased.The financial industry,characterized by strict regulations and supervision,demanded heightened security measures.Consequently,a comprehensive study on prompt injection attacks and a security defense framework was delved into in large language models within the financial sector.A risk taxonomy encompassing eight forms of input prompt injection attacks and five categories of safety scenarios on the output side was developed,and a financial domain large model prompt injection attack dataset,FIN-CSAPrompts,was collected using a human-machine adversarial approach.An end-to-end security defense framework against prompt injection attacks in large language models was proposed and tested,and comparative evaluations were performed using prevalent open-source large language models.The research indicated that in the financial industry,the application of the proposed security defense framework significantly enhanced the defensive capabilities of Chinese large language models,effectively reducing the generation of inappropriate content and improving their resilience against prompt injection attacks.This research provided a reference and foundation for further research on the security issues of Chinese large language models in the financial domain,offering datasets,evaluation metrics,and solutions for consideration and adaptation.

关 键 词：金融大语言模型安全 提示注入 风险分类体系 大模型数据集 法律风险检测 

分 类 号：TP391[自动化与计算机技术—计算机应用技术]