LLM-based classifiers can reduce the proportion of records to screen with minimal loss of relevant studies: a case study

Background and Objective The Epistemonikos Sustainable Knowledge Platform (SKP) integrates a suite of machine learning tools designed to make evidence synthesis more efficient, including access to the Epistemonikos Database of Trials (ED-Trials). One feature is the use of classifiers combined with large language models (LLMs) to support abstract screening during study selection. LLM-based classifiers can be applied to populations, interventions, and study design features. Their primary aim is to reduce the proportion of records needed to be screened by automatically excluding records with a high probability of being ineligible. We aimed to evaluate SKP's LLM-based classifiers on abstract screenings of records retrieved via (scenario 1) a traditional systematic literature search ( original ) and (scenario 2) a search conducted within ED-Trials and subsequently forwarded to SKP. Methods We used data from a completed but at the time unpublished systematic review and network meta-analysis (NMA) as reference standards. We evaluated 29 LLM-based classifier combinations in both scenarios and assessed the proportion of records excluded by automated screening and whether any relevant study was wrongly excluded and therefore missed. To determine the impact of missed studies, we reran the network meta-analyses of the original review, excluding missed studies from the model, and we evaluated how missed studies impact the certainty of evidence ratings. Results LLM-based classifiers excluded by automated screening, 5.8% to 46.6% of 1129 records identified by the original search and 7.7% to 34.3% of 1003 records based on the SKP search. More specific classifiers excluded more records than broader ones. Of the 29 combinations applied to the original search results, 10 incorrectly excluded the same two relevant studies. Based on the SKP review, four combinations missed one study, and 10 combinations missed two studies. However, exclusion of these studies resulted in minimal changes to NMA effect estimates (risk ratio changes of 0.0-0.04 in both scenarios) and did not alter certainty of evidence ratings. Conclusion LLM-based classifiers are a promising strategy for reducing the amount of records to screen while keeping the risk of missing relevant studies low. However, further evaluations using multiple use cases across different medical topics are necessary to learn more about the generalizability of our results. Plain Language Summary When conducting a systematic review, research teams typically have to read thousands of research summaries to find the studies relevant to their review question. We tested whether classifiers powered by large language models (a form of artificial intelligence) could help by automatically filtering out clearly irrelevant studies. We tested these classifiers within the Epistemonikos Sustainable Knowledge Platform, a software that supports systematic review conduct. We used a completed systematic review (that was not published at that time) as our benchmark. We then tested 29 classifiers in two situations: one where studies were gathered using a comprehensive literature search in multiple databases and sources and one where they came from Epistemonikos's own trial database. For each setup, we measured how many studies were filtered out and how many of them were wrongly excluded. When relevant studies were missed, we reran original meta-analysis without them to see if their absence would influence the overall results of the review. Classifiers removed between 6% and 47% of studies from the comprehensive search and 8% to 34% from the Epistemonikos database search. Even when relevant studies were excluded, the estimated treatment effects shifted only slightly and the confidence in the evidence stayed the same. These classifiers look like a useful way to cut down the proportion of research summaries to be read; however, more testing across different topics is needed before we can be confident that the approach works well.