Teaching Doctors in Training to Reason With Artificial Intelligence

Large language models (LLMs) have rapidly entered routine use in medical education andclinical practice. Prior randomized trials have shown that, although LLMs can outperformindividual clinicians on some reasoning benchmarks, providing physicians with access toan LLM does not necessarily improve their diagnostic or management performance, and LLMsalone may perform better than physician‐plus‐LLM teams. These findings suggest thathuman‐AI collaboration is fraught, non‐trivial and may require explicit training.TEACH‐AI is a pragmatic, multi‐site, two‐arm randomized controlled trial embedded withinprotected residency didactic time. The primary objective is to determine whether a singlein‐person workshop on the basics of LLMs and best practices of prompting/verificationstrategies improves residents' performance on an AI‐assisted assessment, compared withresidents who complete the simulation before receiving the workshop. All participantswill ultimately receive the same workshop and the same assessment (either workshop‐firstvs assessment‐first).The simulation consists of multiple fully synthetic vignettes delivered via a passwordprotected LLM interface. Residents interact freely with the LLM using natural‐languageprompts and then submit structured final responses regarding aspects such as leadingdiagnosis, differential, management plan, and/or justification. The platform will recordprompts, model outputs, final answers, and timing. Vignette scoring combines correctnessof the final diagnosis or management plan. Scoring is conducted by blinded faculty usingstandardized rubrics and then any discrepancies will be resolved through multiple roundsof discussions.The trial will enroll up to 200 residents across four ACGME‐accredited programs (internalmedicine at Beth Israel Deaconess Medical Center, Stanford University, and CambridgeHealth Alliance; family medicine at AdventHealth Orlando).