Evaluating a Text-Prompt AI Assistant for Chest CT Scans (AI-REPORT Study)

This study investigates whether an artificial intelligence (AI) system that draftspreliminary radiology reports can help experienced chest CT radiologists work fasterwhile maintaining or improving report quality. The trial is conducted in two sequentialphases. The first phase uses a set of complex, real‐world historical cases. Radiologistsinterpret these cases both with and without the help of the AI‐generated draft(AI‐report) in a controlled, crossover study design. The second phase is a prospective,real‐world deployment where the same AI‐report system is integrated into the clinicalworkflow of participating radiologists as they interpret new, incoming chest CT scans inreal time. We measure the time it takes to complete reports and, through blindedevaluations by other senior doctors, assess the quality of the final reports created withand without AI assistance. The goal is to determine if this AI tool can makeradiologists' work more efficient and support high‐quality patient care in actualpractice.1. Detailed Description1.1 Study DesignThis is a two‐phase, multicenter, multireader, multicase (MRMC) study designed toevaluate the real‐world clinical utility of an AI report generation system (AI‐report). 1. Stage 1 (controlled crossover evaluation): This stage employs a retrospective,randomized, two‐period crossover design. A curated set of complex historical chestCT cases, previously discussed in multidisciplinary team (MDT) meetings, is used.Each participating radiologist acts as their own control, interpreting the samecases both with and without the AI draft under controlled conditions. 2. Stage 2 (prospective real‐world deployment): This stage is a prospective,observational study. The validated AI‐report system is deployed into the liveclinical workflow of the participating radiologists. They use the system inreal‐time as they interpret new, consecutive chest CT scans from their clinicalduties, allowing for evaluation in an authentic clinical environment.1.2 Objectives 1. Primary objectives: To evaluate the impact of the AI‐report system on 1) radiologistefficiency (interpretation time) and 2) the clinical quality of finalized reports,assessed in both a controlled retrospective setting (Phase 1) and a prospectivereal‐world setting (Phase 2). 2. Secondary objectives: To assess the nature and clinical significance of edits madeto AI drafts, and to evaluate system usability and integration into the routinereporting workflow.1.3 Study Population 1. Radiologist Readers: Board‐certified radiologists with ≥ 3 years of independentthoracic imaging practice. 2. Blinded Evaluators: Eleven senior clinicians from the original MDT panels thatcontributed the Phase 1 cases, responsible for blinded quality assessment.1.4 InterventionThe intervention is the provision of a fully AI‐generated draft radiology report(AI‐report). In Phase 1, this is provided within a controlled reading platform forhistorical cases. In Phase 2, the system is integrated into the clinical PictureArchiving and Communication System (PACS)/Radiology Information System (RIS) to generatedrafts for prospective, real‐time cases.2. Study ProceduresPhase 1 (Retrospective Crossover): The 400 historical MDT cases are used. The studyinvolves two reading rounds with a washout period. In each round, radiologists interpreta set of cases, with the AI condition (draft provided or not) randomized and crossed overbetween rounds. Interpretation time is recorded, and all finalized reports are collectedfor blinded pairwise comparison by the evaluator panel.Phase 2 (Prospective Deployment): Following Phase 1, the AI‐report system is activated inthe clinical environment for participating radiologists. During a defined prospectiveobservation period, the system generates drafts for eligible new chest CT scans.Radiologists use these drafts in their daily work. Reporting time and the AI draftsalongside the finalized human‐edited reports are collected for analysis. Report qualityin this phase is assessed longitudinally and through sampling.3. Outcome Measures3.1 Primary Outcomes:Efficiency: Change in median interpretation time per case with vs. without AI‐reportassistance (Phase 1) and the distribution of reporting times during real‐world use (Phase2).Quality: Superiority score from blinded paired comparisons of AI‐assisted vs. unassistedreports (Phase 1). Qualitative and quantitative assessment of report adequacy in theprospective cohort (Phase 2).3.2 Secondary Outcomes:Clinical significance of radiologist modifications to AI drafts (5‐point scale).System usability and workflow integration scores from post‐study surveys.4. Statistical AnalysisAnalysis will account for the MRMC design in Phase 1 using hierarchical models. Phase 2data will be analyzed using descriptive statistics and statistical process controlmethods where appropriate. The two phases will be analyzed separately to provide insightsinto efficacy (Phase 1) and effectiveness (Phase 2).