The first steps in the evaluation of a"black-box"decision support tool:a protocol and feasibility study for the evaluation of Watson for Oncology  

作　　者：Lotte Keikes Stephanie Medlock Daniěl J.van de Berg Shuxin Zhang Onno R.Guicherit Cornelis J.A.Punt Martijn G.H.van Oijen 

机构地区：[1]Department of medical oncology,Cancer Center Amsterdam,Academic Medical Center,University of Amsterdam,the Netherlands [2]Department of medical informatics,Academic Medical Center,University of Amsterdam,Amsterdam,the Netherlands [3]Department of Surgery,University Cancer Center Leiden/the Hague,the Hague,the Netherlands

出　　处：《Journal of Clinical & Translational Research》2018年第3期11-23,共13页临床和转化研究

摘　　要：Background and aim:Medical specialists aim to provide evidence-based care based on the most recent scientific insights,but with the ongoing expansion of medical literature it seems unfeasible to remain updated.”Black-box”decision support tools such as Watson for Oncology(Watson)are gaining attention as they offer a promising opportunity to conquer this challenging issue,but it is not known if the advice given is congruent with guidelines or clinically valid in other settings.We present a protocol for the content evaluation of black-box decision support tools and a feasibility study to test the content and usability of Watson using this protocol.Methods:The protocol consists of developing synthetic patient cases based on Dutch guidelines and expert opinion,entering the synthetic cases into Watson and Oncoguide,noting the response of each system and evaluating the result using a cross-tabulation scoring system resulting in a score range of??12 to+12.Treatment options that were not recommended according to the Dutch guideline were labeled with a”red flag”if Watson recommended it,and an”orange flag”if Watson suggested it for consideration.To test the feasibility of applying the protocol,we developed synthetic patient cases for the adjuvant treatment of stage I to stage III colon cancer based on relevant patient,clinical and tumor characteristics and followed our protocol.Additionally,for the feasibility study we also compared the recommendations from the NCCN guideline with Watson’s advice,and evaluated usability by a cognitive walkthrough method.Results:In total,we developed 190 synthetic patient cases(stage I:n=8;stage II:n=110;and stage III:n=72).Overall concordance scores per case for Watson versus Oncoguide ranged from a minimum score of-4(n=6)to a maximum score of+12(n=17)and from??4(n=9)to+12(n=24)for Watson versus the NCCN guidelines).In total,69 cases(36%)were labeled with red flags,96 cases(51%)with orange flags and 25 cases(13%)without flags.For the comparison of Watson with the NCCN guidelin

关 键 词：evidence-based care decision support tool Watson for Oncology clinical practice guideline quality of care personalized medicine 

分 类 号：R73[医药卫生—肿瘤]