Systematic reviews are a cornerstone of evidence-based medicine, providing a rigorous and comprehensive synthesis of research on a specific topic. They involve identifying, appraising, and synthesizing all relevant studies that meet predefined eligibility criteria to answer a specific research question 1. However, there can be confusion surrounding the terms "studies" and "reports of studies" within the context of systematic reviews. This article aims to clarify the distinction between these two concepts and discuss their implications for the quality and reporting of systematic reviews.
Definitions
Studies
In the context of systematic reviews, a study refers to an investigation that includes a defined group of participants and one or more interventions and outcomes 2. It represents the primary unit of interest and analysis in a systematic review 3. Studies can take various forms, including randomized controlled trials, observational studies, and qualitative studies 4. For example, a study might investigate the effectiveness of a new drug in treating a specific disease, or it might explore the experiences of patients undergoing a particular surgical procedure.
Reports of Studies
A report of a study is any document that provides information about a particular study 5. This can include journal articles, preprints, conference abstracts, study register entries, clinical study reports, dissertations, unpublished manuscripts, government reports, or any other document providing relevant information 2. Importantly, a single study might have multiple reports associated with it 3. For instance, a study might have a published protocol, a journal article presenting the main findings, and a conference abstract summarizing the results.
Differences Between Studies and Reports of Studies
The key difference between studies and reports of studies lies in their level of aggregation. A study represents a single investigation, while a report of a study is a description or summary of that investigation. This distinction is crucial because systematic reviews aim to synthesize evidence from individual studies. However, it is important to recognize that reports of studies are essential for accessing information about the studies themselves 3. For example, reviewers often rely on published articles or reports to extract data and assess the risk of bias in the underlying studies.
To illustrate this difference, consider a clinical trial investigating the effectiveness of a new treatment for hypertension. The trial itself is the study, while the journal article published in a medical journal describing the trial's methodology, results, and conclusions is a report of the study. The same study might also have a conference abstract and a clinical study report submitted to a regulatory agency, each representing a different report of the same study.
Implications for Systematic Reviews
The distinction between studies and reports of studies has several important implications for the conduct and reporting of systematic reviews:
- Identifying and Selecting Studies: Systematic reviewers must carefully distinguish between studies and reports of studies during the literature search and screening process. This involves identifying all relevant reports of a study and then grouping them together to ensure that each study is included only once in the review 6. For instance, a reviewer might identify a journal article, a conference abstract, and a clinical trial registration record, all related to the same study. These reports would need to be grouped together to avoid counting the study multiple times.
- Assessing the Risk of Bias: When assessing the risk of bias in included studies, systematic reviewers should consider the quality of the reporting in addition to the methodological quality of the study itself. Poor reporting can introduce bias and affect the reliability of the study's findings 7. This is because incomplete or inaccurate reporting can obscure important details about the study design, conduct, or analysis, making it difficult to assess the validity of the results.
- Extracting Data: Systematic reviewers need to extract data from all relevant reports of a study to capture the full range of information available. This might involve extracting data from multiple publications, unpublished reports, or even contacting study authors for additional information 8. For example, a reviewer might extract data on the study population and intervention from a published article, while obtaining information on adverse events from a clinical study report.
- When to Include Reports of Studies: Systematic reviews should include all relevant studies, regardless of the language they were published in, to avoid language bias 9. Reviewers should also consider reports of studies when they contribute useful information to the review, even if they are not the primary source of data 10. For example, a study protocol might provide valuable information about the study design and planned analyses, even if the full results have not yet been published.
Impact of Including Reports of Studies on the Quality of Evidence
Including multiple reports of the same study in a systematic review can potentially introduce bias and affect the quality of the evidence synthesized. This is because different reports of the same study might present different aspects of the study or emphasize different findings. For example, a journal article might focus on the primary outcome of a study, while a conference abstract might highlight secondary outcomes or subgroup analyses.
One potential source of bias is publication bias, which arises because studies with statistically significant results are more likely to be published than those with non-significant findings 11. This can lead to an overestimation of the true effect of an intervention when conducting a systematic review.
Another important consideration is the prospective validity of randomized controlled trials (RCTs). Prospective validity refers to the extent to which a study was conducted and reported according to a pre-specified plan 12. Studies that deviate from their original protocols or introduce post-randomization biases can lead to unreliable findings.
The GRADE (Grading of Recommendations Assessment, Development, and Evaluation) framework provides a structured approach for assessing the quality of evidence in systematic reviews 13. GRADE considers five factors that can lower the quality of evidence: risk of bias, inconsistency, indirectness, imprecision, and publication bias. It also considers three factors that can raise the quality of evidence: large effect, dose-response gradient, and all plausible confounding considered.
To mitigate the potential impact of including multiple reports of the same study and ensure the quality of evidence, systematic reviewers should:
- Carefully assess the quality of each report: This involves evaluating the completeness and accuracy of the reporting, as well as the risk of bias in the study itself 14.
- Extract data from all relevant reports: This ensures that all available information is considered and that no important findings are overlooked 15.
- Use appropriate statistical methods: When combining data from multiple reports of the same study, systematic reviewers should use statistical methods that account for the correlation between the reports 14.
- Consider the prospective validity of RCTs: Reviewers should prioritize studies that have been prospectively registered and reported according to their registered protocols 12.
- Apply the GRADE framework: This helps to systematically assess the quality of evidence and provide a transparent and structured approach for grading the strength of recommendations 13.
Methods Used to Identify and Assess Reports of Studies
Systematic reviewers use various methods to identify and assess reports of studies, including:
- Comprehensive search strategies: This involves searching multiple databases, including bibliographic databases, trial registers, and grey literature sources, to identify all relevant reports.
- Citation searching: This involves tracking citations to and from included studies to identify additional reports.
- Contacting study authors: This can be useful for obtaining unpublished reports or clarifying information about the study.
- Using study-based resources: This includes searching resources such as trials registers and trials results databases, which provide information about individual studies rather than individual reports.
- Considering clinical study reports: Organizations like Cochrane encourage the use of data from clinical study reports, especially when the intervention is of high importance and there's a risk of reporting bias 10. These reports often contain more detailed information than published articles and can provide valuable insights into the study conduct and findings.
Reporting Standards for Systematic Reviews
Reporting guidelines, such as PRISMA, provide specific recommendations for reporting information about studies and reports of studies in systematic reviews. These recommendations include:
- Clearly defining the eligibility criteria for studies: This helps to ensure that all relevant studies are included in the review.
- Providing a detailed description of the search strategy: This allows readers to assess the comprehensiveness of the search.
- Reporting the number of studies and reports of studies identified, screened, and included: This provides transparency about the review process.
- Describing the characteristics of included studies: This helps readers to understand the nature of the evidence included in the review.
- Presenting the results of the risk of bias assessment: This allows readers to assess the quality of the included studies.
- Clearly reporting the review methods and findings: Clear reporting allows readers to evaluate the rigor of the methods applied, interpret the findings appropriately, and use the review for healthcare decision-making 16.
Conclusion
The distinction between studies and reports of studies is an important consideration in the conduct and reporting of systematic reviews. By carefully distinguishing between these two concepts and applying rigorous methods for identifying, selecting, and assessing reports of studies, systematic reviewers can ensure that they are synthesizing evidence from individual studies in a comprehensive and unbiased manner. This can help to improve the quality and reliability of systematic reviews, ultimately contributing to better healthcare decision-making. Furthermore, adhering to reporting standards, such as PRISMA, ensures transparency and facilitates the appropriate interpretation and utilization of systematic review findings.
Works cited
1. Systematic Review Defined - LibGuides, accessed January 23, 2025, https://med.cornell.libguides.com/systematicreviews/definitions
2. PRISMA 2020 explanation and elaboration: updated guidance and exemplars for reporting systematic reviews | The BMJ, accessed January 23, 2025, https://www.bmj.com/content/372/bmj.n160
3. 6.1.1.3 Studies versus reports of studies - Cochrane Handbook for Systematic Reviews of Interventions, accessed January 23, 2025, https://handbook-5-1.cochrane.org/chapter_6/6_1_1_3_studies_versus_reports_of_studies.htm
4. Systematic Reviews: Levels of evidence and study design - ECU Library Guides, accessed January 23, 2025, https://ecu.au.libguides.com/systematic-reviews/levels-of-evidence
5. library.svhm.org.au, accessed January 23, 2025, https://library.svhm.org.au/systematic-reviews/records-reports#:~:text=What%20is%20a%20%22report%22%20in,information%20about%20a%20particular%20study.
6. Record vs Report vs Study: Standard Terminology for the PRISMA Flow Diagrams in Systematic Reviews - Farhad Shokraneh, accessed January 23, 2025, https://farhadinfo.medium.com/record-vs-report-vs-study-standard-terminology-for-the-prrisma-flow-diagrams-in-systematic-reviews-93ef6529efa8
7. Systematic Reviews: Step 6: Assess Quality of Included Studies - Subject Research Guides, accessed January 23, 2025, https://guides.lib.unc.edu/systematic-reviews/assess-quality
8. Evaluating - Systematic Reviews - LibGuides at University of Texas School of Public Health, accessed January 23, 2025, https://libguides.sph.uth.tmc.edu/SystematicReviews/Evaluating
9. Should I do a systematic review? - Systematic Reviews - Guides at ..., accessed January 23, 2025, https://guides.dml.georgetown.edu/systematicreviews
10. When to include clinical study reports and regulatory documents in systematic reviews - PubMed, accessed January 23, 2025, https://pubmed.ncbi.nlm.nih.gov/30309870/
11. Any advice on systematic reviews and study selection? - ResearchGate, accessed January 23, 2025, https://www.researchgate.net/post/Any-advice-on-systematic-reviews-and-study-selection
12. Full article: 'Trustworthy' systematic reviews can only result in meaningful conclusions if the quality of randomized clinical trials and the certainty of evidence improves - Taylor & Francis Online, accessed January 23, 2025, https://www.tandfonline.com/doi/full/10.1080/10669817.2024.2377490
13. Examining the quality of evidence to support the effectiveness of interventions: an analysis of systematic reviews | BMJ Open, accessed January 23, 2025, https://bmjopen.bmj.com/content/6/5/e011051
14. Appraising systematic reviews: a comprehensive guide to ensuring validity and reliability, accessed January 23, 2025, https://www.frontiersin.org/journals/research-metrics-and-analytics/articles/10.3389/frma.2023.1268045/full
15. How to review and assess a systematic review and meta-analysis article: a methodological study (secondary publication) - PMC - PubMed Central, accessed January 23, 2025, https://pmc.ncbi.nlm.nih.gov/articles/PMC10449599/
16. Chapter III: Reporting the review - Cochrane Training, accessed January 23, 2025, https://training.cochrane.org/handbook/current/chapter-iii