Epistemonikos database methods

Search methods

The main aim of Epistemonikos is to identify all of the systematic reviews relevant for health decision-making.

The identification of other types of evidence has not been completed yet, so Epistemonikos database should not be considered a comprehensive database of broad synthesis, primary studies or structured summaries (see definitions below).

In order to identify systematic reviews multiple electronic databases and other sources are regularly screened without any restriction, including:

Databases regularly updated:

  1. {{::origin.name}} {{::origin.name}} Last searched: {{::last_insertion.date | date }} - {{::origin.commentary}}

Other search sources

All of the systematic reviews being summarized in structured summaries, broad syntheses, and methodological studies (which is an exclusion criteria from Epistemonikos) are also included in the database.

Additionally, Epistemonikos is a living project. So, hundreds of professionals and researchers are using it to identify the best available evidence for real-life questions. Any time they identify relevant reviews these are assessed and included if eligible.

Criteria for considering articles for Epistemonikos database

We select articles in three main categories plus one secondary category:

Broad synthesis

We group in this category different types of articles aiming to make a synthesis of systematic reviews and, sometimes, primary studies.

We consider a broad synthesis for inclusion if,

  1. Reports an explicit method that includes searching in at least one electronic database.
  2. Its main objective is to synthesize systematic reviews. Primary studies can be searched for to complement the results of reviews.

Synthesis of primary studies (i.e. systematic reviews):

We consider a systematic review for inclusion if it fulfills the following criteria:

  1. Provides a description of at least one eligibility criterion
  2. Its main objective is to synthesize primary studies (other synthesis might be used as an additional source for studies).
  3. Reports an explicit method that includes searching in at least one electronic database.

We also include synthesis of primary studies that do not fulfill the above definition but are judged to add valuable information. Some examples are individual patient meta-analysis or meta-analysis of unpublished data.

Primary studies

Primary study is an umbrella term that includes any study design, qualitative or quantitative, where data is collected from individuals or groups of people.

We read the full text of the articles in the above categories to extract the list of included studies that the authors reported, and we incorporate them to the database. We do not add any additional criteria to those imposed by the researchers conducting the systematic reviews.

Structured summaries

Apart from the 3 main categories we also include a secondary category of structured summaries. Structured summaries are summaries of an article developed for non-research audiences which follow a pre-established format which include at least a critical appraisal of the article.

Exclusion criteria

We exclude articles from any of the above categories if they:

Screening

Deduplication

Articles coming from different databases are deduplicated using our own software followed by a manual revision of borderline cases.

Selection of articles

Potentially eligible articles are initially classified by a machine-learning algorithm. Articles with a high probability of eligibility enter the database as soon as available. So, all of the articles are checked by the network of collaborators, but those with a higher score are assessed first.

Articles excluded by the machine are checked by human screeners, starting by those with a higher machine learning score.

Articles excluded by one screener are classified by the machine learning algorithm too, and those with a high score are checked by a second screener.

Any discrepancy between machine or human, or between different humans is resolved by a human arbiter.

All members of the network of screeners go through a brief training and follow a calibration exercise.

List of studies included in systematic reviews

Collaborators read the selected systematic reviews in full text and extract the list of references of included studies. If a list of included studies is not clearly provided by a systematic review, we extract it from tables, annexes or from the text.

If it is not possible to identify a reliable list, we contact authors by email when possible.

The studies included in systematic reviews that fulfill our inclusion criteria are entered into the database independent of the source where the reviewers found them, language or publication status.

Matrices of evidence

Starting from any systematic review for which the information of included studies has been entered in the database, Epistemonikos software generates a table displaying the cluster of systematic reviews that share at least one included study, and all of the studies included in those reviews.

This table is trimmed by collaborators so only systematic reviews and primary studies answering the question of interest are retained.

After removal of irrelevant articles, the software retrieves any other systematic review sharing included studies with any of the primary studies in the matrix. Relevant articles are added.

A second collaborator reviews the included/excluded articles of the matrix.

After completion of the matrix with the information in Epistemonikos database, cross-citation searches for all of the included primary studies are conducted in Google Scholar, Scopus and Web of Science. Relevant systematic reviews (and their included studies) are entered into Epistemonikos and then in the matrix of evidence.

Epistemonikos software automatically detect any new systematic review sharing primary studies with the matrix. Collaborators are notified and they include/exclude the new evidence as soon as it is assessed.