Quality Assessment Tools and Their Applicability
Quality assessment is a critical step in conducting a systematic review. It involves evaluating the methodological quality of the included studies to assess the risk of bias and the overall confidence in the findings. This ensures that the conclusions drawn from the systematic review are valid and reliable. This article provides a comprehensive guide to conducting quality assessment in systematic reviews, covering various aspects, including the selection of appropriate tools, common biases to look for, and the interpretation and reporting of results.
Several quality assessment tools are available, each with its strengths and weaknesses. Selecting the most appropriate tool depends on the type of studies included in the systematic review and the specific research question. Here's a closer look at some of the widely used tools:
| Tool | Purpose | Target Study Design | Key Considerations |
|---|---|---|---|
| AMSTAR 2 | Critically appraise systematic reviews | Randomized and non-randomized studies of healthcare interventions | Consists of 16 items that evaluate various aspects of the systematic review process, including research question formulation, study selection and data extraction, risk of bias assessment, and statistical analysis. Includes a comprehensive user guide. Can be used for reviews of randomized and non-randomized studies of healthcare interventions. |
| Cochrane Risk-of-Bias 2 (RoB 2) | Assess the quality and risk of bias | Randomized clinical trials in Cochrane systematic reviews | Focuses on different aspects of trial design, conduct, and reporting to identify potential biases. |
| Newcastle-Ottawa Scale (NOS) | Assess the quality of non-randomized studies | Non-randomized studies, including case-control and cohort studies, in meta-analyses | Evaluates the selection of study groups, comparability of groups, and ascertainment of exposure or outcome. Scoring is based on awarding points for selection (maximum 4 points), comparability (maximum 2 points), and outcomes (maximum 3 points). Has been adapted for cross-sectional studies. |
| ROBINS-I | Assess the risk of bias | Non-randomized studies of interventions | Specifically designed for studies that did not use randomization to allocate interventions and helps evaluate the risk of bias in the estimated effects of interventions. It is the preferred tool for non-randomized studies of interventions in Cochrane Reviews, according to the Cochrane Scientific Committee. |
| QUADAS-2 | Assess quality and risk of bias | Systematic reviews of diagnostic accuracy studies | Focuses on four key domains: patient selection, index test, reference standard, and flow and timing. |
Other quality assessment tools include the Critical Appraisal Skills Programme (CASP) checklists, the Centre for Evidence-Based Medicine (CEBM) critical appraisal tools, and the Grading of Recommendations, Assessment, Development, and Evaluations (GRADE) approach. Cochrane also provides Overviews of Reviews, which use systematic methods to search for and identify multiple systematic reviews on related research questions in the same topic area. The PRISMA 2020 statement offers guidelines for reporting systematic reviews, primarily focusing on those evaluating the effects of interventions.
When selecting a quality assessment tool, it's important to consider factors such as the type of study design (e.g., randomized controlled trials, cohort studies, case-control studies), the specific research question, and the resources available. Some tools are more comprehensive than others, and some are more specific to certain types of studies. For example, if the systematic review includes only randomized controlled trials, the Cochrane Risk-of-Bias 2 tool may be appropriate. However, if the review includes a mix of study designs, a more general tool like the Newcastle-Ottawa Scale might be more suitable.
Common Biases in Quality Assessment
During quality assessment, it is essential to be aware of common biases that can affect the validity of study findings. These biases can arise from various sources, including:
- Selection bias: This occurs when the selection of participants in a study is not random, leading to systematic differences between the groups being compared.
- Performance bias: This occurs when there are systematic differences in the care provided to the intervention and control groups, other than the intervention being studied.
- Detection bias: This occurs when there are systematic differences in how outcomes are assessed between the intervention and control groups.
- Attrition bias: This occurs when there are systematic differences in the loss of participants from the study between the intervention and control groups.
- Reporting bias: This occurs when there is selective reporting of outcomes.
Other biases to consider include information bias, measurement bias, and confounding bias. It is important to note that a study can be of high quality but still have a risk of bias. For example, in many situations, it is impractical or impossible to blind participants or study personnel to the intervention group. This does not mean that the study is of low quality, but it does mean that there is a risk of bias resulting from knowledge of the intervention status.
There are many possible biases, including bias from reference material or reference method, bias from the all-method mean of a proficiency testing or external quality assurance survey, bias from the mean of a peer group, bias from a comparison method, bias between identical instruments in the same laboratory, and bias between reagent lots. The concept of commutability is also relevant to bias assessment. Commutability refers to the degree to which the results of a measurement procedure are comparable when applied to different samples or populations. If a measurement procedure is not commutable, it may introduce bias into the results of a study.
Strategies for minimizing bias include blinding, using standardized data collection methods, and ensuring complete follow-up. Blinding involves concealing the allocation of participants to intervention and control groups from the participants, researchers, or both. This can help to prevent performance and detection bias. Standardized data collection methods can help to reduce measurement bias. Ensuring complete follow-up can help to reduce attrition bias.
Software and Tools for Quality Assessment
Several software and tools can assist with quality assessment in systematic reviews. These tools can help automate the process, improve consistency, and reduce the risk of errors. Some of the available tools include:
- RevMan: This software, developed by Cochrane, provides a structured approach to conducting systematic reviews, including quality assessment.
- DistillerSR: This software automates every stage of the systematic review process, including quality assessment.
- Rayyan: This tool helps with screening search results and discarding irrelevant studies, which can be helpful in the initial stages of quality assessment.
Other tools include Covidence, which offers a platform for collaboration and data management in systematic reviews, and the Systematic Review Accelerator, which provides a suite of tools to support the systematic review process.
Interpreting and Reporting the Results of Quality Assessment
Once the quality assessment is completed, the results should be interpreted and reported in a clear and concise manner. This involves summarizing the overall quality of the included studies and the risk of bias for each study. The results can be presented in various formats, such as tables, figures, or narrative summaries.
When interpreting the results, it is important to consider the potential impact of the identified biases on the findings of the systematic review. Studies with a high risk of bias may have overestimated or underestimated the true effect of the intervention. The overall confidence in the findings of the systematic review should be assessed based on the quality of the included studies and the risk of bias. This assessment should consider the severity of the identified biases, the consistency of the findings across studies, and the directness of the evidence.
To present the results of quality assessment effectively, consider using subheadings to organize the information and make it more reader-friendly. For example, you could use subheadings such as "Overall Quality of Included Studies" and "Risk of Bias Assessment." You can also use tables or figures to summarize the results in a visually appealing way.
Furthermore, it's important to consider the use of sensitivity analysis when interpreting the results of quality assessment. Sensitivity analysis can explore the effects of including or excluding certain studies in the meta-analysis, particularly when the risk of bias varies across studies. This can help to assess the robustness of the findings and the potential impact of bias on the overall results.
Examples of Quality Assessment in Published Systematic Reviews
Examining how quality assessment is conducted in published systematic reviews can provide valuable insights and guidance. Here are a few examples:
- A systematic review on the neural effects of music on emotion regulation used the Newcastle-Ottawa Scale to assess the quality of non-randomized studies included in the review.
- A systematic review on the effectiveness of probiotics in reducing eczema symptoms used the Cochrane Risk of Bias tool to assess the quality of randomized controlled trials.
- A systematic review on the association of total white cell count with mortality in patients with peripheral arterial disease used the ROBINS-I tool to assess the risk of bias in non-randomized studies.
- A study used AMSTAR to appraise 42 reviews focused on therapies for gastro-esophageal reflux, peptic ulcer disease, and other acid-related diseases.
These examples demonstrate the variety of quality assessment tools used in different systematic reviews and how the choice of tool depends on the specific research question and the type of studies included.
Conclusion
Quality assessment is an integral part of conducting a systematic review. By critically appraising the methodological quality of included studies, reviewers can ensure that the findings of the systematic review are valid and reliable. Selecting appropriate quality assessment tools, being aware of common biases, and interpreting and reporting the results accurately are crucial steps in this process. Utilizing available software and tools can further enhance the efficiency and rigor of quality assessment in systematic reviews.
However, quality assessment is not without its challenges. The selection of an appropriate tool can be difficult, and the assessment process can be time-consuming and subjective. There is also a need for ongoing development and refinement of quality assessment tools to keep pace with evolving research methods and standards.
Despite these challenges, quality assessment plays a vital role in evidence-based practice. By ensuring the quality of the evidence used to inform healthcare decisions, quality assessment contributes to better patient care and improved health outcomes. Furthermore, it highlights areas where research methods can be improved and where future research is needed.