Zero-shot Thoracic Oncologic History Generation for Radiologists Using Retrieval-augmented Large Language Model Pipeline

Background: Gathering oncologic history is necessary but inefficient. Recent efforts to generate clinical summaries using large language models (LLMs) have relied on manual information retrieval, prompt tuning, and subjective evaluation, limiting clinical translation. Purpose: To develop and evaluate a zero-shot LLM pipeline that programmatically generates structured oncologic histories using retrieval-augmented generation to ground LLM outputs. Materials and Methods: Retrospective electronic health record data were collected for thoracic oncology patients at a cancer center from January 2018 to April 2024. Twenty surveyed radiologists selected 10 clinical parameters for summarization. A retrieval-augmented generation pipeline was developed to filter clinical data. Three Health Insurance Portability and Accountability Act–compliant LLMs—OpenAI’s GPT-4o mini, o3-mini, and GPT-5mini—produced structured summaries using a defined data schema and a zero-shot prompt approach. Summaries were manually evaluated against retrieved clinical text, with completeness reference standard based on the output of GPT-5-mini. Two radiology residents were timed summarizing clinical parameters, establishing a 240-second benchmark. Comparisons between models were conducted using paired t tests for accuracy, completeness, cost, and processing time. Cost-benefit analyses were conducted using manual extraction time, automated summary generation time, and national survey data on radiologist productivity. Results: Of 2433 patients, 50 were randomly selected for LLM summarization (mean age, 67.0 years ± 12.5 [SD]; 32 female patients). The GPT-5mini pipeline achieved the highest mean completeness score of 95.5% (95% CI: 94.2, 96.7) and was similar to o3-mini for a mean accuracy of 97.9% (95% CI: 96.8, 99.1). GPT-4o mini had the fastest processing time per summary, averaging 12.7 seconds compared with the 240-second benchmark (P <.001). Ideally, these time savings project to annual incremental revenue of $19 525 to $70 942 per radiologist, with costs of $30.26 to $32.46. Conclusion: The LLM pipeline, which does not require manual fine-tuning or information retrieval, accurately summarizes oncologic history. The completeness and speed of these summaries vary depending on the LLM used.