Conducting a knowledge synthesis takes careful planning and comprises many steps. These steps may vary depending on the discipline and the type of review being conducted. For example, the scope, search parameters, extent of analysis, and timeline may vary between review types, but in general, reviews follow a common series of steps as outlined below.
Good preparation is essential for the success of any review project. This stage of the review process typically involves several components including the following:
The following resources provide some useful tips to consider when getting started with a knowledge synthesis review.
Parker, R., & Sikora, L. (2022). Literature reviews: Key considerations and tips from knowledge synthesis librarians. Journal of Graduate Medical Education, 14(1), 32–35. https://doi.org/10.4300/JGME-D-21-01114.1
Unity Health Toronto, Library Services. (n.d.). Knowledge synthesis readiness checklist. Retrieved March 21, 2024, from https://guides.hsict.library.utoronto.ca/ld.php?content_id=35293022
University of Toronto Libraries. (2024, April 30). Knowledge syntheses: Systematic & scoping reviews, and other review types: The importance of the team. Retrieved June 18, 2024, from https://guides.library.utoronto.ca/c.php?g=713309&p=5104945
For information and resources regarding question formulation, protocols, searching, etc., please visit the relevant tabs above.
A critical step in the review process is developing a clearly defined, answerable research question. Question frameworks can aid in the construction of an appropriate research question by helping you to focus on the key concepts in your research topic. There are many frameworks to choose from. Some of the more well-known question frameworks are listed below:
PICO Widely used for medical and health-related questions, the acronym stands for:
P = Patient or Population or Problem
I = Intervention
C = Comparison
O = Outcome
Many variants of PICO have been developed for use in other disciplines/contexts. For example:
PICOC (Social sciences questions) = Population, Intervention, Comparison, Outcome, Context
CHIP (Psychology questions) = Context, How, Issues, Population
ECLIPSE (Management/Service delivery questions) = Expectation, Client, Location, Improvement/Innovation, Professionals, Service
SPICE (Qualitative questions) = Setting, Perspective, Intervention/phenomenon of Interest, Comparison, Evaluation
SPIDER (Mixed methods questions) = Setting, Phenomenon of Interest, Design, Evaluation, Research type
Source: (Anderson & Booth, 2022)
The following video from Drill Hall Library (Universities at Medway, UK) provides a brief introduction on how a question framework such as PICO, can help you to focus your research question.
Eligibility Criteria
Once the research question has been defined and prior to beginning the search for research studies, the review team will need to determine inclusion and exclusion criteria. These are the criteria that the studies must meet in order to be included in the review. For example, inclusion/exclusion criteria could encompass elements such as:
The following guide from the University of Melbourne Library, provides more examples of Common Inclusion/Exclusion Criteria.
Anderson, P.F., & Booth, A. (2022). Question frameworks. In M.J. Foster & S.T. Jewell (Eds.), Piecing together systematic reviews and other evidence synthesis (pp. 45–56). Rowman & Littlefield.
University of Maryland Libraries. (2024, May 8). Systematic review: Developing a research question. Retrieved June 18, 2024, from https://lib.guides.umd.edu/SR/research_question
A review protocol is a detailed plan on how the review will be conducted. It will contain several sections including the research question and a rationale for the review, a list of team members, search strategy, inclusion/exclusion criteria, study selection and screening strategy, and data extraction and synthesis methods. The protocol helps to minimize bias during the review process and helps to ensure that the review team stays on track. Once the protocol is completed, it is considered good practice to make it publicly available by registering it in a protocol registry. This will notify other researchers that a review is planned and will help prevent duplication of research. The following resources include selected links to protocol guidelines and templates, and protocol registry databases.
PRISMA-P (Preferred Reporting Items for Systematic Reviews and Meta-Analyses for systematic review protocols)
Chapter II: Planning a Cochrane Review, Section ii.1.4 Cochrane Protocols. In Cochrane handbook for systematic reviews of interventions (Version 6.4, updated August 2023). Cochrane. https://training.cochrane.org/handbook/current/chapter-ii#section-ii-1-4
Chapter 5: Review Protocol. In Campbell systematic reviews: Policies and guidelines (Version 1.8, February 2021). Campbell Collaboration. https://doi.org/10.4073/cpg.2016.1
Collaboration for Environmental Evidence (CEE) Systematic Review Protocol
Evidence Synthesis Protocol Template created by Kate Ghezzi-Kopel and Jaron Porciello (Open Science Framework)
JBI manual for evidence synthesis (2024 Edition). JBI. https://jbi-global-wiki.refined.site/space/MANUAL
How to write a scoping review protocol: Guidance and template (Open Science Framework)
Review Protocol Template created by Sarah Visintini
PROSPERO (International prospective register of systematic reviews)
Cochrane Reviews and Protocols
INPLASY (International Platform of Registered Systematic Review and Meta-analysis Protocols)
Note: Some journals also publish review protocols. For example, the journal Systematic Reviews publishes systematic review products including systematic review protocols.
Protocols for Systematic and Scoping Reviews (video by Carrie Price, Towson University, Albert S. Cook Library)
The foundation of any knowledge synthesis is the search of the research literature. The extent of the search may vary depending on the topic and the type of review being conducted, but typically the search will be extensive and the methods used will be transparent and well-documented so that the search strategies are reproducible by others.
Preliminary - a preliminary search, sometimes called a scoping or exploratory search, is usually conducted during the planning and protocol development phase of a knowledge synthesis. There are many reasons to conduct a preliminary search, including the following:
Comprehensive/Exhaustive - A comprehensive or exhaustive search of the literature will involve searching selected bibliographic databases (more than one) to identify as many peer-reviewed publications as possible that relate to the research question. The search may also include searching for conference proceedings, government reports, theses, or other types of grey literature (materials produced by non-commercial publishers}. Searching may be further augmented by handsearching selected journals that are not indexed in databases, browsing reference lists of relevant studies retrieved, cited reference searching, and contacting authors of studies.
Sources: (Booth et al., 2022; Fowler & Jewell, 2022)
The following video, Fundamentals of Systematic Literature Searching, from Conhrane Austria, provides an overview of the essential elements of a systematic literature search.
To ensure that search strategies are transparent and reproducible, it is essential to maintain a detailed record of searches including keywords and controlled vocabulary used, databases searched, dates searches were run, limits or restrictions applied, numbers of records retrieved, and other search methods used. This information will need to be reported in the methodology section of the review. The PRISMA-S (Preferred Reporting Items for Systematic reviews and Meta-Analyses search extension) checklist provides researchers with guidance on what items should be reported.
When searching in bibliographic databases, it is advisable to create an account in each database searched. This will allow you to save, edit, and rerun searches, and to set up search alerts. Consult the Library's Search and Journal Alerts tutorial for more information.
Searching databases for systematic reviews and other types of knowledge syntheses may result in hundreds or thousands of citations. However, the number of citations that can be exported at one time varies between databases. If needing to export large numbers of citations, you may have to export your citations in batches (e.g., 1-500, 501-1000, etc.). Auburn University Libraries Systematic Review guide provides some direction on how to export large numbers of citations from several well-known databases.
Citation management software such as Zotero, Endnote, and Mendeley are useful tools for storing, organizing, and de-duplicating search results. Please consult the Library's Citation Styles guide for more information on citation managers.
Locating Existing Reviews:
Systematic or other review types are often published in peer reviewed journals. These reviews can be found by searching bibliographic databases for the review topic and adding the phrase "systematic review" (or "scoping review", meta-analysis, etc.) to your search. Other places to search include systematic review databases or repositories of completed reviews, and protocol registries of reviews in progress, such as the following:
Campbell Systematic Reviews - topic areas include crime & justice, education, international development, social welfare and more. Browse by subject categories or article types (e.g., systematic review, protocol, etc.).
Collaboration for Environmental Evidence - evidence synthesis relating to environmental policy and management. Maintains the CEE Evidence Synthesis Library, a list of CEE Evidence Syntheses in Progress; and the CEE Database of Evidence Reviews (CEEDER), a searchable database of syntheses (e.g. critical reviews, meta-analyses, systematic reviews, rapid reviews) that includes an assessment of the reliability of each synthesis.
Cochrane Library (available from Databases A-Z) - contains Cochrane systematic reviews (including overviews of reviews and rapid reviews) relating to healthcare, as well as Cochrane protocols.
Evidence for Policy and Practice Information Centre (EPPI Centre) – maintains a library of EPPI Centre reviews. Coverage includes systematic reviews in the fields of education, health promotion and public health, as well as social welfare and international development.
PROCEED - an open access registry of titles and protocols for prospective evidence syntheses in the environmental sector.
PROSPERO - an international database of prospectively registered systematic reviews in health and social care, welfare, public health, education, crime, justice, and international development, where there is a health-related outcome.
Repository of Systematic Reviews on Interventions in Environment, Climate Change and Health (World Health Organization) - aims to highlight the state of evidence on interventions in selected environment, climate change, and health (ECH) topics that improve health outcomes or reduce harmful exposures, and to identify knowledge and evidence gaps in ECH. Downloadable database includes systematic reviews, meta-analyses, and overviews of systematic reviews.
3ie Development Evidence Portal (International Initiative for Impact Evaluation) – a repository of rigorous evidence on what works in international development containing high-quality impact evaluations, systematic reviews, and evidence gap maps.
Search Tips:
Atkinson, L.Z., & Cipriani, A. (2018). How to carry out a literature search for a systematic review: A practical guide. BJPsych Advances, 24(2), 74–82. https://doi.org/10.1192/bja.2017.3
Booth, A., Sutton, A., Clowes, M., & Martyn-St James, M. (2022). Systematic approaches to a successful literature review (3rd ed.). Sage.
Bramer, W.M. et al. (2018). A systematic approach to searching: An efficient and complete method to develop literature searches. Journal of the Medical Library Association, 106(4), 531–541. https://doi.org/10.5195/jmla.2018.283
Fowler, S.A., & Jewell, S.T. (2022). Database Searching. In M.J. Foster & S.T. Jewell (Eds.), Piecing together systematic reviews and other evidence synthesis (pp. 45–56). Rowman & Littlefield.
Hirt, J. et al. (2024). Guidance on terminology, application, and reporting of citation searching: The TARCiS statement. BMJ, 385, e078384. https://doi.org/10.1136/bmj-2023-078384
Kugley, S. et al. (2017, February). Searching for studies: A guide to information retrieval for Campbell systematic reviews (Version 1.1). Campbell Collaboration. https://doi.org/10.4073/cmg.2016.1
Searching for systematic reviews: Introduction - Tutorial
Grey Literature:
Grey Literature (University of Toronto Libraries Research Guide)
Paez, A. (2017). Gray literature: An important resource in systematic reviews. Journal of Evidence-Based Medicine, 10(3), 233–240. https://doi.org/10.1111/jebm.12266
Peer Reviewing Guideline (for librarians and information specialists):
McGowan, J., Sampson, M., Salzwedel, D.M., Cogo, E., Foerster, V., & Lefebvre, C. (2016). PRESS peer review of electronic search strategies: 2015 guideline statement. Journal of Clinical Epidemiology, 75, 40-46. https://doi.org/10.1016/j.jclinepi.2016.01.021
See also companion document: PRESS – Peer Review of Electronic Search Strategies: 2015 Guideline Explanation and Elaboration
(PRESS E&E). Ottawa: CADTH; 2016 Jan.
Documenting Search Strategies:
PRISMA-S (PRISMA-S: PRISMA Search Reporting Extension)
Rethlefsen, M.L., Kirtley, S., Waffenschmidt, S. et al. (2021). PRISMA-S: An extension to the PRISMA Statement for reporting literature searches in systematic reviews. Systematic Reviews, 10, Article 39. https://doi.org/10.1186/s13643-020-01542-z
See also Hirt, J. et al. under Search Tips (above) for guidance on how to report citation searching.
Once searching is completed, the collected studies will need to be screened for eligibility. Comprehensive searching across multiple databases often results in the retrieval of multiple copies of some studies. These duplicate studies will need to be removed first. Citation management software such as Zotero, EndNote, or Mendeley, or specialized evidence synthesis screening tools, such as Covidence, can be used to deduplicate search results.
After duplicate records have been removed, the titles and abstracts of records are screened using agreed upon inclusion and exclusion criteria as defined during the development of the review protocol. Studies that don't meet the inclusion criteria at this stage are weeded out and the full-text of the remaining studies is obtained and then screened using the inclusion and exclusion criteria. Decisions made during the screening process need to be documented including numbers of studies included and excluded. The Knowledge Syntheses Screening for Articles guide provided by the Gerstein Science Information Centre at the University of Toronto Libraries, provides additional details on how to screen search results.
Depending on the type and size of the evidence synthesis being conducted, more than one reviewer may be required for screening studies. For systematic reviews, a minimum of two independent reviewers is recommended (Cochrane).
Chapter 4.6: Selecting studies. In Cochrane Handbook for Systematic Reviews of Interventions (Version 6.4, updated August 2023). Cochrane. https://www.training.cochrane.org/handbook
PRISMA Flow Diagram (for documenting the screening process)
Selected Subscription and Free Screening Tools:
Subscription/Fee-based:
Covidence (web-based software for the production of systematic reviews and other types of knowledge syntheses)
DistillerSR (literature review software that supports systematic and other review types)
EPPI-Reviewer (web-based software tool for all types of systematic review including meta-analysis, framework synthesis and thematic
synthesis)
JBI SUMARI (software to assist in the conduct of JBI systematic reviews)
PICO Portal (web-based tool for citation screening)
Rayyan (web-based screening tool; free to get started with subscription options)
RevMan (web-based platform for managing Cochrane Reviews; free for authors working on Cochrane reviews)
Free:
Abstrackr (web-based tool developed by Center for Evidence Synthesis in Health, Brown University)
Colandr (web-based, open access platform for conducting evidence reviews)
An important component of systematic reviews and other knowledge syntheses is an assessment or critical appraisal of included studies for relevance, quality, and risk of bias. The level and degree of appraisal will depend on the type of review being conducted. Some review types, scoping reviews for example, typically do not require a critical appraisal.
There are a number of tools and checklists available for appraising research evidence including the following resources:
Critical Appraisal Skills Programme (CASP) Checklists
SIGN (Scottish Intercollegiate Guidelines Network)
Quality Assessment and Risk of Bias Tool Repository
Data extraction is the process of extracting key information from the studies that have been selected for inclusion in the review. Data is typically collected using a data extraction form or table. Forms can be developed from scratch or existing forms can be adapted. Several of the systematic review tools listed in the Screening section of this guide, include data extraction forms/templates (e.g., Covidence, Colandr, DistillerSR, JBISUMARI).
Collected data is then combined and analyzed. Various methods may be used to synthesize the data including qualitative and quantitative/statistical (e.g., meta-analysis) synthesis methods. The evidence synthesis organizations (e.g., Cochrane and others) provide guidance on data extraction including what data to collect and how to synthesize collected data.
For more information, please consult the following selected resources:
SRDR+ (Systematic Review Data Repository)
Cochrane handbook for systematic reviews of interventions (Version 6.4, updated August 2023). Cochrane. https://www.training.cochrane.org/ handbook.
Guidelines and standards for evidence synthesis in environmental management (Version 5.1, 2022). Collaboration for Environmental Evidence. https://environmentalevidence.org/information-for-authors/
Elamin, M.B. et al. (2009). Choice of data extraction tools for systematic reviews depends on resources and review complexity. Journal of Clinical Epidemiology, 62, 506–510. https://doi.org/10.1016/j.jclinepi.2008.10.016
Levett, P. (2024, June 10). Systematic reviews: Data extraction/coding/study characteristics/results. Himmelfarb Health Sciences Library. Retrieved June 25, 2024, from https://guides.himmelfarb.gwu.edu/c.php?g=27797&p=170447
Pollock, D. et al. (2023). Recommendations for the extraction, analysis, and presentation of results in scoping reviews. JBI Evidence Synthesis, 21(3), 520–532. https://doi.org/10.11124/JBIES-22-00123
The final step in an evidence or knowledge synthesis project is writing and disseminating the review. The format or structure that the review takes will depend on a number of factors, including the purpose of the review and the type of review being conducted. There are various reporting standards or guidelines available that provide guidance in what information should be included in the review write-up. A reporting guideline is "a checklist, flow diagram, or structured text to guide authors in reporting a specific type of research, developed using explicit methodology.” Equator Network
PRISMA (Preferred Reporting Items for Systematic reviews and Meta-Analyses) is a common reporting standard, and there are several PRISMA extensions including PRISMA-ScR (Preferred Reporting Items for Systematic reviews and Meta-Analyses Extension Statement for Scoping Reviews). Journals that publish evidence syntheses may recommend specific reporting guidelines and some journals may produce their own guidelines. Be sure to use a reporting guideline that is appropriate for the type of review you have undertaken and if planning to publish your review in a particular journal, check to see what reporting guideline the journal prefers.
PRISMA (Preferred Reporting Items for Systematic reviews and Meta-Analyses)
ROSES (RepOrting standards for Systematic Evidence Syntheses)