Effects of double data extraction on errors in evidence synthesis: a crossover, multicenter, investigator-blinded, randomized controlled trial

ObjectivesThe objective was to investigate the role of double extraction in reducing data errors in evidence synthesis for pharmaceutical and non-pharmaceutical interventions. DesignCrossover randomized controlled trial (RCT). SettingUniversity teaching center and hospital evidence-based medicine center. ParticipantsEligible 100 participants were 2nd year or above post-graduate students (e.g., masters, doctoral program), who were randomly (1:1) assigned for data extraction tasks of either 10 RCTs of pharmaceutical interventions or 10 of non-pharmaceutical interventions, followed by a cross-over pattern and a further double-checking process. InterventionThe intervention of this trial was double-checking process for data extraction. Primary and secondary outcome measuresThe primary outcome was the error rates for RCTs in the pharmaceutical versus non-pharmaceutical intervention group, in terms of both study level and cell level (2 by 2 table). The secondary outcome was the absolute difference in the error rates before and after the double-checking process for both the pharmaceutical and non-pharmaceutical intervention groups, again, in terms of both study level and cell level (2 by 2 table). ResultsThe error rates in RCTs of pharmaceutical and non-pharmaceutical groups were 64.65% and 59.90%, with an absolute difference of 4.75% and an odds ratio (OR) of 1.29 [95% confidence interval (CI): 1.06 to 1.57, P = 0.01] when measured at the study level. After double-checking, the error rates decreased to 44.88% and 39.54%, and the difference between the two groups remained at 5.34%, with the OR of 1.27 (95%CI: 1.1 to 1.46; P < 0.01). Similar results were observed when measured at the cell level. ConclusionDouble-checking reduced data extraction errors, but the error rate still remained high after the process. Further evidence synthesis research may consider to use triple data extraction or else effective methods to minimize potential errors. Trial registration numberChinese Clinical Trial Registry Center (Identifier: ChiCTR2200062206) Strengths and limitations of the studyThis is the third randomized trial focusing on data extraction strategies and the first one in the Asia-Pacific region. This is the first randomized trial that compares error rates of data extraction in trials of pharmaceutical interventions and non-pharmaceutical interventions. We validated the effectiveness of double data extraction while also identifying its limitations, providing valuable evidence for future data extraction strategies. To ensure the feasibility of the trial, we restricted subjects to 2 post-graduate students or above, which may affects the representativeness of the sample. The readability of the chosen randomized controlled trials for pharmaceutical and non-pharmaceutical interventions may vary, potentially resulting in selection bias that can distort the outcomes.