Evaluating the accuracy and reproducibility of ChatGPT-4 in answering patient questions related to small intestinal bacterial overgrowth  

作　　者：Lauren Schlussel Jamil S Samaan Yin Chan Bianca Chang Yee Hui Yeo Wee Han Ng Ali Rezaie 

机构地区：[1]Division of Gastroenterology and Hepatology,Cedars-Sinai Medical Center,Los Angeles,CA 90048,United States [2]Bristol Medical School,University of Bristol,BS81TH,Bristol,United Kingdom [3]Medically Associated Science and Technology Program,Cedars-Sinai Medical Center,Los Angeles,CA 90048,United States

出　　处：《Artificial Intelligence in Gastroenterology》2024年第1期14-21,共8页胃肠病学中的人工智能（英文）

摘　　要：BACKGROUND Small intestinal bacterial overgrowth(SIBO)poses diagnostic and treatment challenges due to its complex management and evolving guidelines.Patients often seek online information related to their health,prompting interest in large language models,like GPT-4,as potential sources of patient education.AIM To investigate ChatGPT-4's accuracy and reproducibility in responding to patient questions related to SIBO.METHODS A total of 27 patient questions related to SIBO were curated from professional societies,Facebook groups,and Reddit threads.Each question was entered into GPT-4 twice on separate days to examine reproducibility of accuracy on separate occasions.GPT-4 generated responses were independently evaluated for accuracy and reproducibility by two motility fellowship-trained gastroenterologists.A third senior fellowship-trained gastroenterologist resolved disagreements.Accuracy of responses were graded using the scale:(1)Comprehensive;(2)Correct but inadequate;(3)Some correct and some incorrect;or(4)Completely incorrect.Two responses were generated for every question to evaluate reproducibility in accuracy.RESULTS In evaluating GPT-4's effectiveness at answering SIBO-related questions,it provided responses with correct information to 18/27(66.7%)of questions,with 16/27(59.3%)of responses graded as comprehensive and 2/27(7.4%)responses graded as correct but inadequate.The model provided responses with incorrect information to 9/27(33.3%)of questions,with 4/27(14.8%)of responses graded as completely incorrect and 5/27(18.5%)of responses graded as mixed correct and incorrect data.Accuracy varied by question category,with questions related to“basic knowledge”achieving the highest proportion of comprehensive responses(90%)and no incorrect responses.On the other hand,the“treatment”related questions yielded the lowest proportion of comprehensive responses(33.3%)and highest percent of completely incorrect responses(33.3%).A total of 77.8%of questions yielded reproducible responses.CONCLUSION Though GPT

关 键 词：Small intestinal bacterial overgrowth MOTILITY Artificial intelligence Chat-GPT Large language models Patient education 

分 类 号：R57[医药卫生—消化系统]