Quality Control in Continuous Updates for Living Systematic Reviews

Living systematic reviews (LSRs) are becoming increasingly important for providing up-to-date evidence syntheses, especially in rapidly evolving fields. Unlike traditional systematic reviews, which are conducted once and rarely updated, LSRs are continually updated to incorporate new research as it becomes available. This allows them to provide the most current and reliable evidence for decision-making in healthcare, policy, and research. However, the continuous updating process of LSRs presents unique challenges for maintaining quality control. This article will discuss the importance of quality control in LSRs, outline the challenges and best practices for maintaining quality in continuous updates, and explore the potential of automation and machine learning to improve quality control processes.

The Importance of Quality Control in Living Systematic Reviews

The value of LSRs lies in their ability to provide reliable and current summaries of evidence. However, this reliability hinges on rigorous quality control measures throughout the review process. High-quality LSRs are essential for:

• Informing clinical practice: Healthcare professionals rely on LSRs to stay informed about the latest evidence and make well-informed decisions about patient care.
• Guiding policy development: Policymakers use LSRs to develop evidence-based policies and guidelines.
• Supporting research: Researchers use LSRs to identify knowledge gaps and inform the design of future studies.

Maintaining quality in LSRs is crucial to ensure that these stakeholders can trust the evidence and use it effectively to improve healthcare outcomes, shape policies, and advance research.

Challenges of Maintaining Quality in Continuous Updates

The dynamic nature of LSRs presents several challenges for quality control:

• Time constraints: Frequent updates require efficient processes to incorporate new evidence quickly without compromising quality. Studies have shown that there is often a substantial delay between the availability of study results and their inclusion in systematic reviews. For example, an analysis of neurotrauma systematic reviews found a median time of 2.5 to 6.5 years between the publication of primary study results and their inclusion in a systematic reviewmedian time of 2.5 to 6.5 years.
• Resource limitations: LSRs require ongoing resources for searching, screening, data extraction, and synthesis. Cochrane reviews, for instance, often take more than 12 months to complete and are infrequently updatedCochrane reviews.
• Maintaining consistency: Ensuring consistency in methods and reporting across updates can be challenging, especially with changes in the review team or evolving methodologies. The decision to update an LSR depends not only on the evolving evidence base but also on the specific review question and the resources available to the review team.
• Bias: The continuous influx of new studies can introduce bias if not managed carefully.
• Evolving evidence: The interpretation of evidence may change as new studies emerge, requiring careful reassessment and potential revisions to conclusions.
• Quality of evidence: Different factors can affect the quality of evidence included in LSRs. These factors include the risk of bias in individual studies, the precision of effect estimates, the indirectness of the evidence to the review question, inconsistency in findings across studies, and the potential for publication bias.

Addressing these challenges requires a systematic approach to quality control that incorporates best practices, utilizes available tools and technologies, and adapts to the evolving nature of the evidence base.

Best Practices for Quality Control in Continuous Updates

Several best practices can help ensure the quality of LSRs:

• Establish a clear protocol: A detailed protocol should outline the review question, eligibility criteria, search strategy, data extraction methods, and quality assessment criteria. This protocol should be registered in a publicly accessible database, such as PROSPERO, to ensure transparency and accountabilityPROSPERO.
• Use robust search strategies: Comprehensive search strategies should be developed to identify all relevant studies, including those published in languages other than English and in grey literature sources. The search strategy should be regularly updated to capture new studies as they become availablesearch strategy.
• Employ rigorous quality assessment: Appropriate quality assessment tools should be used to evaluate the risk of bias in included studies. The choice of tool should be based on the study design and the specific research question.
• Ensure data integrity: Data extraction should be performed by at least two independent reviewers to minimize errors. Standardized data extraction forms and clear coding guidelines should be used to ensure consistencydata extraction.
• Maintain a clear audit trail: All decisions and changes made throughout the review process should be documented to ensure transparency and facilitate future updates. This includes documenting any changes to the protocol, search strategy, inclusion criteria, or data extraction methodsaudit trail.
• Communicate updates clearly: Changes made in each update should be clearly communicated to users, including any new studies, revised conclusions, or changes in methodologycommunicate updates.

Quality Control Checklists and Tools

Several quality control checklists and tools can be used to assess the quality of LSRs and their updates. These can be categorized into checklists and software tools.

These checklists and software tools can help ensure that LSRs are conducted and updated according to rigorous methodological standards.

Automation and Machine Learning for Quality Control

Automation and machine learning have the potential to improve quality control in LSRs significantly. These technologies can assist with various tasks, including:

• Study selection: Machine learning algorithms can be trained to identify relevant studies from databases and screen out irrelevant ones, reducing the workload for reviewersstudy selection.
• Data extraction: Automated tools can extract data from included studies, minimizing errors and saving timedata extraction.
• Risk of bias assessment: Machine learning models can be developed to assess the risk of bias in included studies, improving consistency and efficiencyrisk of bias assessment.
• Meta-analysis: Automated tools can perform meta-analyses, reducing the risk of errors and allowing for rapid updatesmeta-analysis.

However, it is important to note that these technologies are still under development and have limitations. For example, research has shown that none of the currently available tools cover the whole living review workflow while providing complete and transparent validation of their automation methodslimitations of automation. In particular, challenges remain in developing tools that can effectively link related studies and synthesize evidence between updateslink related studies. Furthermore, some quality dimensions, such as indirectness, inconsistency, and publication bias, are more challenging to automate than otherspublication bias.

Despite these limitations, automation and machine learning hold great promise for improving the efficiency and accuracy of quality control in LSRs. As these technologies continue to evolve, they are likely to play an increasingly important role in ensuring the reliability and timeliness of LSRs.

While automation offers promising solutions, the human element of quality control remains crucial, particularly in the form of peer review.

Peer Review in Quality Control

Peer review plays a crucial role in ensuring the quality of LSRs. Independent experts in the field review the LSR protocol and updates to assess the rigor of the methodology, the validity of the conclusions, and the clarity of reporting. Peer reviewers can identify potential biases, methodological flaws, and areas for improvementpeer review improvements. This process helps build trust in science by ensuring that research is conducted to a high standardtrust in science.

In the context of LSRs, peer review may involve:

• Protocol review: Reviewing the protocol before the review begins to ensure its methodological rigor and feasibility.
• Update review: Reviewing updates to assess the quality of the new evidence incorporated and the validity of any revised conclusions.
• Ongoing review: Providing ongoing feedback and guidance to the review team throughout the LSR process.

Peer review also serves as a risk mitigation strategy for ethical breaches in research. Reviewers carefully evaluate whether studies adhere to established ethical guidelines and ensure that research involving human subjects or animals has been conducted with integrityethical breaches. Furthermore, peer review plays a role in assessing the quality of manuscripts before publication, helping to ensure that only high-quality research is disseminatedhigh-quality research.

By engaging independent experts in the review process, LSR authors can enhance the credibility and trustworthiness of their reviews.

Managing Conflicts of Interest

Conflicts of interest (COI) can arise in LSRs when reviewers have financial or personal interests that could influence their judgment. Managing COI is essential to maintain the integrity and credibility of the reviewintegrity and credibility.

Strategies for managing COI in LSRs include:

• Disclosure: Requiring all reviewers to disclose any potential COIdisclosure.
• Exclusion: Excluding reviewers with significant COI from participating in the reviewexclusion.
• Independent oversight: Having an independent panel oversee the review process to ensure objectivityindependent oversight.
• Transparency: Clearly reporting any COI in the LSR publicationtransparency.

By implementing these strategies, LSR authors can minimize the risk of bias and maintain the trustworthiness of their reviews.

Conclusion

Living systematic reviews are a valuable tool for providing current and reliable evidence syntheses. However, maintaining quality in continuous updates requires a multi-faceted approach to quality control. This approach should include adhering to best practices, such as establishing a clear protocol, using robust search strategies, and employing rigorous quality assessment methods. Utilizing quality control checklists and tools, such as AMSTAR 2 and ROB 2, can help ensure that LSRs are conducted and updated according to established standards.

Furthermore, exploring the potential of automation and machine learning can significantly improve the efficiency and accuracy of quality control processes. While these technologies are still under development, they offer promising solutions for tasks such as study selection, data extraction, and risk of bias assessment.

The human element of quality control remains crucial, particularly in the form of peer review. Peer reviewers provide valuable feedback and guidance throughout the LSR process, helping to ensure the rigor of the methodology, the validity of the conclusions, and the ethical conduct of the research.

Finally, managing conflicts of interest is essential to maintain the integrity and credibility of LSRs. By implementing strategies such as disclosure, exclusion, and independent oversight, LSR authors can minimize the risk of bias and ensure that their reviews provide trustworthy evidence for decision-making.

In conclusion, a comprehensive approach to quality control is essential for ensuring the reliability and trustworthiness of living systematic reviews. By combining human expertise with technological advancements, LSR authors can provide high-quality evidence syntheses that inform clinical practice, guide policy development, and support research.