Introduction
Systematic reviews and meta-analyses are essential tools for synthesizing research evidence and informing healthcare decisions. To minimize bias and errors in methodology, these reviews typically involve multiple reviewers independently screening titles, abstracts, and full texts against predefined eligibility criteria1. A crucial step in this process is the selection of relevant studies. However, disagreements between reviewers during screening are inevitable. This can be due to human error, differences in interpretation, or even unconscious biases2. This article explores the challenges associated with reviewer disagreements in systematic reviews, discusses common causes and methods for resolving disagreements, and highlights best practices for minimizing discrepancies and ensuring the rigor and reliability of study selection.
Clear guidelines and best practices are essential for managing disagreements between reviewers effectively. These guidelines should outline the process for resolving conflicts, ensuring transparency and consistency in decision-making. Some key recommendations include:
- Establish a clear protocol: Before commencing the screening process, establish a detailed protocol that outlines the specific steps for handling disagreements. This protocol should specify the methods for resolving conflicts, such as discussion, third-party adjudication, or consensus-based approaches3.
- Maintain blinding: During the initial stages of screening, reviewers should be blinded to each other's decisions to minimize bias. Blinding can be maintained by using software that hides reviewers' votes until a final decision is made. It's important to maintain blinding until a final decision is made and a vote is agreed upon3.
- Facilitate communication: Encourage open communication and discussion between reviewers to resolve disagreements. This can involve face-to-face meetings, online discussions, or email exchanges3.
- Use a third reviewer: In cases where disagreements persist despite discussion, a third reviewer can be brought in to provide an independent assessment and break the tie. The third reviewer should be experienced in systematic reviews and familiar with the inclusion and exclusion criteria3. Discrepancies can be settled between the Content Specialists themselves or by a third party6.
- Document the process: Maintain a detailed record of all disagreements, the methods used to resolve them, and the final decisions made. This documentation ensures transparency and accountability in the study selection process7.
Utilize software tools: Software tools like EviSynth can assist in managing disagreements and maintaining blinding during the screening process. These tools provide features for collaborative screening, conflict identification, and resolution tracking, enhancing the efficiency and transparency of the process.
Disagreements between reviewers can arise from various factors, including:
- Ambiguous eligibility criteria: Unclear or poorly defined inclusion and exclusion criteria can lead to different interpretations and inconsistencies in study selection8.
- Reviewer bias: Reviewers' prior knowledge, beliefs, or experiences can unconsciously influence their judgments and lead to biased screening decisions1.
- Study heterogeneity: Variations in study design, population characteristics, or outcome measures can make it challenging to apply eligibility criteria consistently2.
- Lack of reviewer experience: Inexperienced reviewers may be less familiar with the nuances of systematic reviews and more prone to making errors in study selection, which can lead to disagreements9.
- Inadequate training: Insufficient training on the review protocol and eligibility criteria can contribute to inconsistencies in screening decisions10.
Several methods can be employed to resolve disagreements between reviewers:
- Discussion and consensus: Reviewers can discuss their differing interpretations of the eligibility criteria and attempt to reach a consensus. This approach encourages collaboration and shared decision-making5.
- Third-party adjudication: A third reviewer, often a senior researcher or expert in the field, can be consulted to provide an independent assessment and resolve the disagreement5.
- Arbitration: In some cases, an independent arbitrator may be appointed to make a binding decision on the inclusion or exclusion of a study. (source needed)
- Voting: Reviewers can vote on the inclusion or exclusion of a study, with the majority vote determining the final decision.
- Delphi methodology: This method involves a structured process of consensus-based discussion and anonymized voting, which can be used to achieve agreement among reviewers11.
It's important to note that disagreements can arise not only on whether to include or exclude a study but also on the reasons for exclusion, especially during full-text screening3. The chosen method for resolving disagreements should be clearly documented in the review protocol.
| Method | Advantages | Disadvantages |
|---|---|---|
| Discussion and consensus | Encourages collaboration, shared understanding | May be time-consuming, potential for unresolved disagreements |
| Third-party adjudication | Provides independent assessment, expertise | Requires additional resources, potential for bias from the third reviewer |
| Arbitration | Ensures a final decision | Can be costly, may not consider all perspectives |
| Voting | Simple, efficient | May not be suitable for complex disagreements |
| Delphi methodology | Structured approach, minimizes individual bias | Can be time-consuming, requires careful planning |
The third reviewer plays a critical role in resolving disagreements and ensuring the objectivity of the study selection process. Their responsibilities include:
- Independent assessment: The third reviewer should independently review the study in question and apply the eligibility criteria without being influenced by the initial reviewers' decisions9.
- Breaking ties: When the initial reviewers cannot reach a consensus, the third reviewer provides a deciding vote to resolve the disagreement7.
- Providing expertise: The third reviewer may offer their expertise and knowledge to clarify ambiguous criteria or provide guidance on challenging cases12.
- Ensuring consistency: The third reviewer helps ensure consistency in the application of eligibility criteria across all studies12.
It is crucial that the third reviewer is knowledgeable in both the subject matter and the review protocol to effectively resolve conflicts12. This expertise allows them to make informed decisions and ensure that the final selection of studies aligns with the objectives of the systematic review.
Well-defined inclusion and exclusion criteria are crucial for minimizing disagreements and ensuring the validity of a systematic review. The inclusion/exclusion of studies determines the scope and validity of systematic review results13. Clear criteria provide a framework for reviewers to make consistent and objective decisions about study selection. Key considerations for developing robust criteria include:
- Specificity: Criteria should be specific and unambiguous, leaving little room for interpretation8.
- Relevance: Criteria should be directly relevant to the research question and scope of the review14.
- Justification: Provide a clear rationale for each inclusion and exclusion criterion8.
- Pilot testing: Test the criteria on a small sample of studies to identify potential ambiguities or inconsistencies10.
- Consider potential bias: Be mindful of the potential for bias introduced by inclusion/exclusion criteria. For example, limiting studies to those published in English may exclude valuable research in other languages15. Each criterion should be carefully considered and justified to minimize bias and ensure the inclusivity of the review.
- Document exclusion reasons: When excluding a study, clearly document the primary reason for exclusion. This helps maintain transparency and allows for a better understanding of the decision-making process10.
Pilot testing the screening process is a valuable strategy for identifying and addressing potential areas of disagreement before full-scale screening begins. Pilot testing involves:
- Selecting a sample: Choose a representative sample of studies from the retrieved records. This sample should include articles that are likely eligible, likely ineligible, and those where eligibility is more difficult to determine10.
- Independent screening: Have two or more reviewers independently screen the sample using the eligibility criteria10.
- Assessing agreement: Calculate the level of agreement between reviewers and identify any discrepancies10.
- Refining criteria: Based on the pilot test results, revise the eligibility criteria or provide further clarification to reviewers10. This may involve modifying the criteria in the data extraction form to ensure that the most relevant information is captured16.
Pilot testing not only helps minimize disagreements but can also lead to time and effort savings in the long run by identifying and addressing potential issues early in the review process17.
Disagreements between reviewers are an inherent challenge in the study selection process for systematic reviews. However, by implementing clear guidelines, using appropriate methods for resolving conflicts, and prioritizing well-defined eligibility criteria, researchers can minimize discrepancies and ensure the rigor and reliability of their reviews. Pilot testing the screening process and fostering open communication between reviewers are essential strategies for identifying and addressing potential areas of disagreement.
Unresolved disagreements can have significant implications for the overall conclusions of a systematic review18. Discrepancies in study selection can introduce bias, affect the generalizability of findings, and ultimately reduce confidence in the review's conclusions. Therefore, it is crucial to have robust strategies in place to manage and minimize disagreements.
Ultimately, effective management of reviewer disagreements contributes to the transparency, validity, and trustworthiness of systematic reviews, strengthening their role as a cornerstone of evidence-based healthcare. Further research and development of strategies for minimizing and managing disagreements are essential for continuous improvement in this area. This may involve exploring alternative conflict resolution methods, developing more sophisticated software tools, and providing more comprehensive training for reviewers.
1. What is the rationale of having two or more individuals complete the systematic review?, accessed on January 20, 2025, https://www.researchgate.net/post/What_is_the_rationale_of_having_two_or_more_individuals_complete_the_systematic_review
2. Error rates of human reviewers during abstract screening in ..., accessed on January 20, 2025, https://pmc.ncbi.nlm.nih.gov/articles/PMC6959565/
3. How to resolve conflicts during screening - Covidence, accessed on January 20, 2025, https://www.covidence.org/blog/how-to-resolve-conflicts-during-screening/
4. Resolving Conflicts - Rayyan for Systematic and Scoping Reviews ..., accessed on January 20, 2025, https://research.library.gsu.edu/c.php?g=1059580&p=7702414
5. library.svhm.org.au, accessed on January 20, 2025, https://library.svhm.org.au/systematic-reviews/screen#:~:text=Resolving%20conflicts%20in%20screening&text=Common%20methods%20include%3A,conflict%20until%20agreement%20is%20reached
6. Conducting Systematic Reviews - LibGuides - OHSU, accessed on January 20, 2025, https://libguides.ohsu.edu/systematic-reviews/how-to
7. Screen - Systematic reviews - LibGuides at St. Vincent's Hospital ..., accessed on January 20, 2025, https://library.svhm.org.au/systematic-reviews/screen
8. Define Inclusion/Exclusion Criteria - Systematic Reviews for Social Sciences - LibGuides at University of South Florida Libraries, accessed on January 20, 2025, https://guides.lib.usf.edu/systematicreviews/includeexclude
9. An introduction to the Third Reviewer Model - Editor Resources, accessed on January 20, 2025, https://editorresources.taylorandfrancis.com/peersupport/introduction-third-reviewer-model/
10. Screening Sources - Systematic Reviews - Subject Guides at ..., accessed on January 20, 2025, https://guides.lib.byu.edu/systematicreviews/screeningsources
11. Methodology of Systematic Reviews and Recommendations - PMC - PubMed Central, accessed on January 20, 2025, https://pmc.ncbi.nlm.nih.gov/articles/PMC3143415/
12. Optimizing your systematic review: how many reviewers do you need? - Traverse Science, accessed on January 20, 2025, https://traversescience.com/review/optimizing-your-systematic-review-how-many-reviewers-do-you-need/
13. Selecting studies for systematic review: Inclusion and exclusion criteria - ASHA Journals, accessed on January 20, 2025, https://pubs.asha.org/doi/pdf/10.1044/cicsd_33_s_21
14. Eligibility Criteria - Knowledge syntheses: Systematic & Scoping ..., accessed on January 20, 2025, https://guides.library.utoronto.ca/c.php?g=713309&p=5088425
15. 4. Apply Inclusion and Exclusion Criteria - Systematic Review Overview - LibGuides, accessed on January 20, 2025, https://guides.hsict.library.utoronto.ca/c.php?g=699108&p=4964044
16. Routine piloting in systematic reviews—a modified approach? - PMC - PubMed Central, accessed on January 20, 2025, https://pmc.ncbi.nlm.nih.gov/articles/PMC4108964/
17. Results from the evaluation of the pilot living systematic reviews: - Cochrane Community, accessed on January 20, 2025, https://community.cochrane.org/sites/default/files/uploads/inline-files/Transform/201905%20LSR_pilot_evaluation_report.pdf
18. pubs.asha.org, accessed on January 20, 2025, https://pubs.asha.org/doi/pdf/10.1044/cicsd_33_s_21#:~:text=The%20concept%20of%20inclusion%20and,for%20the%20disorder%20under%20consideration.