Automating Systematic Literature Reviews: A Best Evidence Synthesis of Current Methods

A systematic review is a rigorously and systematically constructed type of literature review that involves the collection and critical analysis of multiple research studies or papers. To understand a systematic review vs literature review, consider that unlike narrative reviews, which offer a broad overview with narrative reporting and subjective analysis of existing literature, SLRs employ a thorough methodology to synthesize all relevant studies on a specific research question, ensuring objectivity and minimizing bias. Furthermore, when considering systematic review vs meta-analysis, a structured literature review like a systematic review uses a predefined search strategy to identify relevant studies, critically appraises their quality, and synthesizes the findings in a systematic and reproducible manner. This rigorous approach makes SLRs a reliable source of evidence for informing decision-making in healthcare, policy, and various other fields. SLRs, supported by frameworks and methodologies from organizations like the Cochrane Collaboration, are widely acknowledged as the gold standard for evidence synthesis, providing the most robust and reliable information for decision-making.

SLRs typically involve the following steps, often guided by established frameworks for systematic review:

Defining the research question

Formulating a clear and focused question that the review aims to answer. This often involves using frameworks like PICO systematic review (Population, Intervention, Comparison, and Outcome) to clearly define the scope of the review. For example, a research question might focus on the effects of exercise programs on glycemic control in type 2 diabetes patients, comparing those who participate in exercise programs with those who do not.

Developing a protocol

Establishing a predefined set of methods for conducting the review. The protocol outlines key aspects, including search strategies, inclusion/exclusion criteria, data extraction procedures, and even considerations for what time frame should be used for a systematic review, ensuring a systematically constructed approach.

Searching for studies

Systematically searching relevant databases and sources to identify all potentially eligible studies. This may involve using Boolean operators (AND, OR, NOT) to refine search results and ensure that all relevant studies are captured.

Screening studies

Evaluating the identified studies based on the inclusion/exclusion criteria to select those that meet the review's requirements. This step often involves assessing the quality of the papers using a specific checklist to ensure that only high-quality studies are included in the review. This crucial stage is essentially data screening, ensuring only pertinent research progresses to the next phases.

Extracting data

Systematically collecting relevant data from the included studies.

Assessing the risk of bias

Critically appraising the methodological quality of the included studies to assess the risk of bias.

Synthesizing the findings

Combining and analyzing the findings of the included studies to answer the research question. This often involves presenting the synthesized data in various formats, such as graphs, tables, and flowcharts, to facilitate understanding and interpretation.

Reporting the results

Presenting the findings of the review clearly and transparently.

Automation can be applied to various stages of the SLR process to improve efficiency and reduce workload. Some key areas where automation can be particularly helpful include:

1. Search

Automation can help develop search strategies and identify relevant studies. Tools like Litsearchr, an R package, can facilitate semi-automatic search strategy development. Polyglot Search can translate search strategies across different databases, while Yale MeSH Analyzer can help identify relevant MeSH terms. Additionally, tools like the Methods Wizard can help researchers write the systematic review protocol using semi-automation. Researchers can also leverage semantic keywords and databases to find relevant papers, using tools that employ semantic search capabilities to identify studies related to a specific topic or concept.

2. Screening

Automated tools can help streamline the process of screening studies for inclusion. One of the major challenges in systematic reviews is the screening process, particularly title and abstract screening, which can be time-consuming and mentally demanding. Tools like EviSynth, Rayyan, and Abstrackr use machine learning to assist with screening and deduplication of citations. EviSynth is designed to use large language models (LLMs) to compare each article with the criteria set for the projects. The "AI Insights" feature provides an LLM decision of whether to include or exclude an article with a percentage of AI confidence in the decision. ASReview utilizes active learning to prioritize studies for screening. These platforms potentially reducing screening time by up to 95%. This can incorporate blind screening and text screening processes.

3. Data Extraction

Automation can help extract data from included studies. RobotReviewer, for example, can automatically extract data from randomized controlled trials (RCTs), including PICO elements, study design, and risk of bias information. DistillerSR offers AI-powered data extraction capabilities, simplifying the process of collecting and organizing study characteristics and outcomes.

4. Risk of Bias Assessment

Automated tools can help assess the risk of bias in included studies. RobotReviewer and EviSynth for example, can automatically generate risk of bias tables based on the Cochrane Risk of Bias tool.

5. Evidence Synthesis

While complete automation of evidence synthesis remains a challenge, AI-powered tools can assist in summarizing and analyzing study findings. For example, ChatGPT can be used to generate summaries of research findings and identify potential themes and patterns in the data. However, it is important to acknowledge the limitations of ChatGPT in article extraction.

While automation offers significant potential for streamlining the SLR process, it is essential to acknowledge its limitations and challenges:

Accuracy and reliability

Automated tools may not always be accurate or reliable, and their performance can vary depending on the specific tool and the data being analyzed. Researchers need confidence in the outputs but must remember that automated systems require human oversight and validation is crucial to ensure the quality of automated outputs. It's important to systematically analyze the results produced by these tools.

Bias

Automated tools can be susceptible to bias, particularly if the training data used to develop the tools are biased. This bias can manifest in various ways, such as the exclusion of relevant studies from certain populations or with specific methodologies. Careful selection of training data and ongoing evaluation of tool performance are essential to mitigate bias.

Complexity

Some automation tools may require specialized skills or knowledge to use effectively. A thorough tool review and understanding of the software's functionalities are necessary before implementation. Researchers must be aware of the limitations of the tools and use them appropriately.

Ethical considerations

The use of automation in SLRs raises ethical considerations, such as the potential for job displacement and the need for transparency and accountability in automated decision-making. Concerns regarding the accuracy of AI-generated content are paramount, as inaccurate information can have serious consequences in medical and healthcare contexts. Additionally, the responsible use of AI requires careful consideration of data and algorithmic bias, the longevity and sustainability of tools, their transparency, reliability, and validity, and fairness and equity.

The field of SLR automation is rapidly evolving, with ongoing research and development of new tools and techniques. Future directions include:

Improved accuracy and reliability

Researchers are working to improve the accuracy and reliability of automated tools through the use of advanced machine learning and natural language processing techniques. This includes improving AI performance in data extraction and risk of bias assessment.

Enhanced transparency

There is a growing emphasis on transparency in the development and use of automated tools, including the publication of training data and algorithms.

Integration with other tools

Researchers are exploring ways to integrate automation tools with other software and platforms used in the SLR process, such as reference managers and statistical analysis software.

Ethical guidelines

There is a need for the development of ethical guidelines for the use of automation in SLRs to ensure responsible and ethical practices.

Automation technologies offer significant potential for streamlining the SLR process, improving efficiency, and reducing workload. This can benefit various stakeholders, including researchers, clinicians, and policymakers. Automation can help researchers save time and resources in conducting a literature review systematic review, clinicians access the latest evidence synthesis faster, and policymakers make more informed decisions grounded in systematic review literature. However, it is crucial to acknowledge the limitations and challenges of automation and use these tools responsibly and ethically. Despite the potential of automation to aid in a comprehensive systematic analysis, human oversight remains crucial to ensure the accuracy, reliability, and ethical use of AI in systematic reviews. With ongoing research and development, automation is poised to play an increasingly important role in the future of SLRs, enabling researchers to conduct more systematic analyses and rigorous reviews while keeping pace with the growing body of scientific literature.