生成式大语言模型在医学考试题库建设中的实践探索  

Exploratory practice of generative large language models in the construction of medical item banks

作　　者：江哲涵 奉世聪 王维民 Jiang Zhehan;Feng Shicong;Wang Weimin(Institute of Medical Education,Peking University,Bejing 100191,China;Graduate School of Education,Peking University,Beijing 100871,China)

机构地区：[1]北京大学医学教育研究所,北京100191 [2]北京大学教育学院,北京100871

出　　处：《中华医学教育杂志》2024年第8期561-569,共9页Chinese Journal of Medical Education

基　　金：国家卫生健康委员会人才交流服务中心项目(202110-335);国家自然科学基金委员会青年科学基金项目(72104006);国家卫生健康委员会国家医学考试中心"十四五"改革重点项目(2022-21)。

摘　　要：传统的医学考试题库建设耗时长且依赖于命题专家资源,而大语言模型为题库建设带来了新方式,其试题生成质量很大程度上取决于提示词的设计。为了提高医学试题质量,帮助医学教师有效利用大语言模型开展命题工作,本文介绍了大语言模型中常用的提示工程,并以"术后胆漏"试题生成为例,探索了零样本、少样本、思维链、自洽性思维链、思维树提示工程策略的命题效果。分析结果显示,零样本和少样本提示操作简便,但在试题多样性和深度上存在一定局限。通过增加思维成分的提示策略,可以引导大语言模型执行草稿、打磨、比较和确定等命题过程,从而提高试题质量。同时,虽然通过改进提示词可以有效提高命题效果,但其具体实施与设计仍有极大的挖掘空间,需要进一步的研究和探索。Item development in healthcare profession education is time-consuming and heavily reliant on content experts.While large language models(LLMs)introduce a new approach to reduce the burdens,the output quality is largely contingent upon the prompt.This article aims to guide educators in effectively leveraging LLMs for item development,enhancing the quality through prompt engineering.Using″postoperative bile leakage″as an example,the paper demonstrates the effectiveness of various prompt engineering strategies,including Zero-shot,Few-shot,Chain of Thought(CoT),CoT with Self-Consistency(CoT-SC),and Tree of Thoughts(ToT).It is found that while Zero-shot and Few-shot methods are straightforward,they have certain limitations in terms of item diversity and depth.Conversely,prompt strategies incorporating″Thought″elements can navigate the LLMs through stages of drafting,refining,comparing,and finalizing,thereby elevating question quality.Although refining prompts indeed leads to notable improvements in question formulation efficacy,there remains substantial room for exploring and optimizing prompt formulations and strategies to further augment the quality of generated questions.The pursuit of advancing prompt engineering techniques holds the promise of significantly elevating the standards of question bank development within medical education.

关 键 词：人工智能 生成式大语言模型 提示工程 医学试题 题库建设 考试命题 

分 类 号：R-4[医药卫生] TP18[自动化与计算机技术—控制理论与控制工程]