What do you think are the most useful methods of ensuring suitable and useful search results when Utilising databases for research?

What is a systematic review?

A systematic review is a method of answering a question by finding, appraising and synthesising evidence in a systematic and unbiased way.

This guide will take you through the steps involved in conducting a systematic review. Click one of the stages below to see information, tips, resources and support available for each part of the process.

Preparation

(average timeline - 5 months*)

Before you start

determining if a systematic review is the right fit and an overview of the logistics in conducting a review.

Define research question

scoping your topic, types of research questions and writing a focused question for your review to answer.

Conducting reviews

(average timeline - 10 months*)

Perform searches

collecting the evidence, recording your search, and exporting to a reference manager.

Screen results

determining which studies to include in the review and methods for reporting on decisions.

Write and publish

(average timeline - 3 months*)

*this is a rough estimate of the time needed to complete each stage, but will vary depending on the complexity of your review.

Systematic Review Service Charter

The Library provides tiered support for researchers conducting systematic reviews. Postgraduate students who conduct systematic reviews as part of their degree can request up to 3 hours of support per systematic review. Postdoctoral researchers and beyond can opt for more extensive support, depending on their requirements and our capacity. For details, please refer to the service charter. If you are interested in Tier 3 (Premium) support, please submit a request and the relevant team will review it.

Feedback

We are always improving our resources. If you'd like to share your feedback with us, please do so via our feedback form.

Before you start

If you haven't conducted a systematic review before, you might be wondering where to start. This section takes you through a series of questions to consider before starting a systematic review.

Is my question appropriate for a Systematic Review?

The systematic review process works best for a clearly defined question with a narrow scope and a single answer. Traditionally, systematic reviews in health disciplines investigate the effectiveness of an intervention (or compare the effectiveness of two interventions), but the methodology is continually adapting to address different question types and disciplines.

A systematic review may not be the best method for your research if your question is too broad or the existing evidence is too sparse or varied. There are other review types that share some of the systematic review's characteristics, namely scoping reviews and systematic literature reviews. Explore differences and similarities of various review types in this comparative table: Types of Reviews.

Complete this online module to find out which review type is the most suitable for your research question:

Scoping reviews

In many cases, the most appropriate review type for a literature review for a PhD thesis would be a scoping review.

To learn about scoping review methodology, refer to Additional Reading section on this page.

What type of systematic review is the most appropriate for my question?

If you are certain that your question can be answered by conducting a systematic review, you need to consider what type of systematic review suits it best.

All systematic reviews share the same methodology. However, different types of questions require different considerations.

To learn what kind of systematic review is the most applicable to your question, complete this online module

Is there an existing systematic review on my topic?

Before starting work on a systematic review check if a systematic review on a similar topic has already been published (or commenced).

Use the following tools to perform a quick search for your topic (a few main keywords will do), and limit to "reviews" or "systematic reviews" (depending on the interface):

• Multidisciplinary databases like Scopus and Web of Science
• The Cochrane Library, especially the Reviews, Trials, and Protocols tabs
• TRIP
• Epistemonikos
• PROSPERO (to find systematic reviews that are already in progress)

Do I have enough time to complete a systematic review?

Systematic reviews take time. The estimated timeline for a Cochrane systematic review is 18 months full time.

Doing a systematic review properly can take months, even if you're not conducting a Cochrane-style systematic review.

If you expect that time might be an issue, consider conducting a rapid review instead.

For more on rapid reviews, refer to Additional reading section on this page.

Do I have access to necessary resources?

A team is an important factor for a successful systematic review. Your team may include:

• Subject matter experts (like your supervisor or colleagues) for question formulation and writing up findings
• A librarian for guidance around literature searching
• Colleagues to become second/third reviewers

If you are planning to perform a meta-analysis, you will also need access to:

• Statistical support for data analysis and synthesis.

Finding collaborators can be tricky. You might consider:

• Reaching out to your network/s to identify people with common research interests
• Using academic social networks like Mendeley and ResearchGate to find potential collaborators
• Keeping your researcher profiles (e.g., USYD Academic Profile, Google Scholar, ORCiD, Scopus ID, and Web of Science Researcher ID) up to date so others can easily find you

Depending on your topic, you might want to explore professional networks to identify potential collaborators among a range of stakeholders including clinicians, patients, customers, or anyone else whose contribution to the review might be useful.

Systematic Review Guides

Most systematic reviews are currently conducted in fields like health and medicine. Increasingly, systematic reviews are being published in other disciplines such as humanities and education.

Throughout this guide, you might need to consult with Cochrane, the Joanna Briggs Institute and the Campbell Collaboration guides for more specific advice. These organisations are responsible for the development of many best practice frameworks and guidelines.

The Joanna Briggs Institute reviewer’s manual (2020)
Contains practical and theoretical guidance on how to conduct a variety of systematic reviews answering different types of questions (e.g. aetiology and risk, qualitative evidence, prevalence and incidence).

Systematic reviews in practice by Crawford, Boyd, & Jonas (2015)
Provides simple, straightforward guidance for conducting a systematic review in health and medicine. Covers the entire systematic review process, from question formulation to publication.

Cochrane handbook for systematic reviews of interventions (2019)
An in-depth guide to conducting systematic reviews. Use this handbook to get clarification on steps and stages you're unclear about. The Cochrane handbook also includes information about MECIR, Cochrane's own set of reporting standards.

How to do a systematic literature review in nursing: a step-by-step guide by Bettany-Saltikov (2012)
Provides practical guidance for conducting a systematic review in Nursing, but is useful for all areas of study. Explains how to complete the different stages of a systematic review in practical terms. Chapter 2 is particularly helpful if you're stuck on question formulation.

Doing a systematic review: a student's guide by Boland, Cherry, & Dickson (2014)
An easy to follow, practical guide on conducting a systematic review as part of a thesis. Breaks each stage of the systematic review process into actions.

Campbell Collaboration
The Campbell Collaboration is an international social science research network that produces high quality, open and policy-relevant evidence syntheses, plain language summaries and policy briefs.

Need help?

To learn what levels of library support are available for researchers conducting systematic reviews explore the SR Service Charter.

The library runs regular workshops to get you started with your literature review. To see upcoming workshops and to register, check the Library Calendar.

Additional Reading

• Munn, Z., Peters, M. D., Stern, C., Tufanaru, C., McArthur, A., & Aromataris, E. (2018). Systematic review or scoping review? Guidance for authors when choosing between a systematic or scoping review approach. BMC medical research methodology, 18(1), 143.
• Peters, M. D. J., Godfrey, C. M., Khalil, H., McInerney, P., Parker, D., & Soares, C. B. (2015). Guidance for conducting systematic scoping reviews. International journal of evidence-based healthcare, 13(3), 141-146. doi:10.1097/xeb.0000000000000050.
• Dobbins, M. Rapid Review Guidebook. Steps for Conducting a Rapid Review. National Collaborating Centre for Methods and Tools.
• Tricco A.C., Langlois E.V., Straus S.E., editors. Rapid reviews to strengthen health policy and systems: a practical guide. Geneva: World Health Organization; 2017. Licence: CC BY-NC-SA 3.0 IGO.

Define research question

A focused research question is imperative for the success of your systematic review. This section provides tips on how to make your research question suitable for a systematic review.

Research topic vs a review question

A 'research topic' is the area of study that you are researching, while a 'review question' is the more focused question that your systematic review aims to answer. Depending on your starting point, you may arrive at a review question by taking different routes.

If you are working in an Applied Sciences field, or as a clinician, your review question might be informed by a problem or a scenario encountered in the lab or in practice.

If you are working towards your PhD, your thesis topic will be much broader than a question for a systematic review. Often, to account for the many topics covered in your thesis, you may need to consider writing a scoping review (for more, see Before you start. For example, your thesis might be about studying a range of non-therapeutic treatments for dementia. In your scoping review you will identify all existing treatments, and in your systematic review you might focus on the effectiveness of only one treatment:

Example

Research topic and scoping review (broad)

• Non-therapeutic treatment of depression

Systematic review question (narrow)

• How effective are music-based therapy programs for relieving depression in patients with Alzheimer's?

Focussing Research Question

To help you focus your research question we suggest exploring different mnemonics. Mnemonics highlight different concepts that you need to focus on. To explore different mnemonics developed for different disciplines refer to the Additional literature section on this page.

In health and medicine, mnemonics might differ depending on the type of question you are working on, e.g., aetiology, prognostic, prevalence, etc. You can explore different mnemonics in this article

For more information on mnemonics refer to Ready for Health Research Module 2.

Why PICO?

The most widely used mnemonic for systematic reviews is PICO – Population, Intervention, Comparator, Outcomes. PICO can be used to formulate a focused research question for systematic reviews in any field where the objective is to assess the effectiveness of an intervention.

PICO originated from evidence-based practice where it is used to help clinicians identify the most important elements of a question when analysing a patient’s condition and suggesting the best treatment.

For example, if your research topic was to explore a range of non-therapeutic treatments for depression, you could use PICO to concentrate on the specific conditions, treatments, and their outcomes:

Example

• Population: patients with Alzheimer's experiencing depression
• Intervention: music-based therapy programs
• Comparison: no comparison
• Outcome: efficacy

You can learn about evidence-based practice and the use of PICO by exploring this EBP subject guide.

For more information on how to use PICO refer to Ready for Health Research Module 2.

Limitations of PICO

When applying PICO to your research topic, remember:

• Use PICO to focus on what already exists in your topic; do not invent concepts just to match all the letters in the mnemonic. If you do not intend to compare anything to your intervention, do not include a comparator.
• If your question is not centred on an intervention or if the concepts do not fit into PICO, your research question might be better suited to a different mnemonic.
• PICO helps you identify main concepts in your topic; it is not designed to help create a search strategy. To learn more about developing a search strategy, see Plan search strategy and select databases section.

For more information on how to translate main concepts into searchable concepts, refer to Ready for Health Research Module 2.

Develop and register protocol

Writing a protocol is an important step in minimising potential bias in the systematic review process. This section provides tips on how to prepare and register a protocol, and define eligibility criteria.

Why do I need to write a protocol?

A protocol specifies the question a systematic review will attempt to answer and the methods that will be used to answer it. Having a protocol is essential because:

• It ensures that the entire team is aware of and in agreement with the question, the methods, the timelines, and the search strategy
• It enables communication with external stakeholders when seeking their contribution
• It keeps you focused and on topic

Preparing a protocol

There are many reporting standards available which outline the essential information you need to provide in a protocol. Requirements will change depending on where you publish the protocol.

Some of these templates include:

Checklist - PRISMA-P
A useful checklist that lists every item you might need to address in a protocol. The accompanying elaboration and explanation document explains each item and provides clear examples.

Templates: Campbell Collaboration
Campbell Collaboration publish systematic reviews in social science, including areas such as crime and justice, education, nutrition, & social welfare. They provide a template for protocols which explains each section in detail.

Template text for reporting search methods - Cochrane
Contains best practice examples from the Cochrane Information Specialist community on how to write about the proposed search in Cochrane protocols. Although the examples are specific to Cochrane protocols, the language and terminology can be adapted to suit your purpose.

SYRCLE
Systematic Review Centre for Laboratory Animal Experiemntation (SYRCLE) developes methodology for systematic reviews in animal studies and provides a template for a protocol.

Amending a protocol

Sometimes reviewers need to change their original protocol after a review has commenced. Any changes made should be reported and explained in an amendment to the protocol, as well as in the systematic review itself (Gough, Oliver, & Thomas, 2017). All registers have procedures for submitting amendments.

Your search strategy needs to be included as part of your protocol

Refer to Plan search strategy and select databases section for more details.

Why do I need to register a protocol?

You must have a protocol even if you are not planning to register it. Registering a protocol is highly recommended for the following reasons:

• To let other researchers know that you are working on a particular topic and help avoid duplication of effort
• To avoid publication bias whereby only the research that rendered positive results is published
• To help readers assess the quality of the review in terms of bias, through comparison of the protocol and the published systematic review

Registering a protocol

You have a range of platforms to choose from when deciding where to register a protocol. Many of the platforms will have specific conditions and limitations that you need to be aware of; refer to the Note section for useful tips.

PROSPERO
An international register of systematic review protocols in the following areas:

• Health and social care
• Welfare
• Public health
• Education
• Crime and justice
• Health-related international development

Note: PROSPERO does not register scoping reviews and does not accept submissions from students.

Cochrane Reviews
Cochrane is a global network of researchers who produce Cochrane Reviews - these reviews are considered the ‘gold standard’ in systematic reviews in the field of health and medicine. Cochrane publish the protocols of Cochrane affiliated researchers through the Cochrane Database of Systematic Reviews

Note: when you register your protocol with Cochrane you commit to the Cochrane-style systematic review. Your topic would need to be approved by one of the 52 Cochrane Review Groups. You will be asked to update your review every two years.

Joanna Briggs Institute
JBI is a not-for-profit research and development centre within the Faculty of Health and Medical Sciences at the University of Adelaide, with more than 70 Collaborating Entities across the world.

Note: You can register your systematic review title with JBI, including scoping review titles, if at least one of the authors belongs to an "affiliated entity". The JBI journal JBI Evidence Synthesis charges US$1000 upon acceptance of full protocols if none of the authors of the submission are members of the JBI Collaboration.

Campbell Collaboration
An international research network that produces systematic reviews of the effects of social interventions, Campbell registers protocols for reviews in multiple areas of study but only if you plan on publishing your systematic review through the Campbell Collaboration. Any interested researcher can propose a review title, form an author team, and submit a proposal. There is a rigorous peer assessment process for the title and, if title is accepted, for the protocol. Refer to the Methodological Expectations of Campbell Collaboration Intervention Reviews (MECCIR) for further details.

Note: Campbell will not publish protocols of reviews that are to be published elsewhere.

Alternative Platforms

You can submit the review protocol as a preprint on one of the following platforms that will provide you with a time stamp indicating that the protocol was created prior to starting the review:

Sydney eScholarship (SES)
SES is the University of Sydney’s open access repository that accepts works from HDR students and academics, but not coursework students.

Open Science Framework
Here is a link to the help guides on creating a registration or preregistration. The OSF will accept protocols from students.

Medrxiv

Research Square

FigShare

Collaborative Approach to Meta-Analysis and Review of Animal Data from Experimental Studies

Eligibility (inclusion and exclusion) criteria

Eligibility criteria, also referred to as inclusion and exclusion criteria, are essential for systematic reviews. They are defined at the start of the systematic review process (a priori) to help you select the most reliable evidence as part of the screening process. Eligibility criteria determine which studies will be considered eligible for inclusion in the final analysis.

Because your eligibility criteria must not change in the process of conduction the review, define these criteria carefully and in a way that allows enough flexibility without introducing additional bias.

Eligibility criteria constitute part of the protocol. Some criteria may include:

• Population characteristics – such as population problem, age range or gender
• Study design or characteristics - such as study design, setting, or time frame
• Report characteristics - including years included, language, and publication status

Eligibility criteria may include features that you want either in or out of the studies considered relevant. You can choose the most convenient way of formulating a criterion either as an inclusion, or as an exclusion. You do not need to match each criterion with its opposite, i.e. include papers in English and exclude papers in all other languages.

Inclusion criteria examples

• Randomised controlled trials
• Papers published in the last 15 years
• Papers in English language

Exclusion criteria examples

• Papers focusing on other types of dementia (e.g. lewy body disease, vascular dementia, alcohol related dementia)
• Papers about other interventions (e.g. art therapy)

Each additional criterion introduces a potential bias into your research. It's important to justify each criterion in the methodology section of your systematic review. Different reporting guidelines may require different criteria.

Need help?

Preparing a protocol requires the entire systematic review team’s effort. Make sure that you consulted subject experts such as your supervisor. Honours students, Higher Degree Research students and staff members can consult an Academic Liaison Librarian for assistance with search strategies.

Additional Reading

• Gough, D., Oliver, S., & Thomas, J. (2017). An introduction to systematic reviews. Los Angeles: SAGE.
• Moher, D., Liberati, A., Tetzlaff, J., Altman, D.G., & The PRISMA Group. (2009). Preferred reporting items for systematic reviews and meta-analyses: The PRISMA statement. PLoS Medicine, 6(7), e1000097.
• Moher, D., Shamseer, L., Clarke, M., Ghersi, D., Liberati, A., Petticrew, M., Shekelle, P., Stewart, L., & the PRISMA-P Group. (2015). Preferred reporting items for systematic review and meta-analysis protocols (PRISMA-P) 2015 statement. Systematic Reviews, 4(1).
• National Institute for Health Research. (n.d.). What is registration? Retrieved from https://www.crd.york.ac.uk/prospero/#aboutpage.
• Patino, C.M., & Ferreira, J.C. (2018).Inclusion and exclusion criteria and why they matter. Jornal Brasileiro de Pneumologia, 44(2), 84.
• PRISMA. (n.d.). What is a protocol? Retrieved from http://prisma-statement.org/Protocols/.
• Shamseer, L., Moher, D., Clarke, M., Ghersi, D., Liberati, A., Petticrew, M., Shekelle, P., Stewart, L.A., & the PRISMA-P Group. (2015). Preferred reporting items for systematic review and meta-analysis protocols (PRISMA-P) 2015: elaboration and explanation. BMJ, 349, g7647.
• Stewart, L., Moher, D., & Shekelle, P. (2012). Why prospective registration of systematic reviews makes sense. Systematic Reviews, 1(7).

Plan search strategy and select databases

Systematic reviews aim to evaluate all relevant information about a topic to answer a question. This section provides tips on what constitutes a comprehensive search and which information sources you need to consider.

What constitutes a comprehensive search?

A comprehensive search has three distinct characteristics:

• It utilises multiple synonyms for each main concept
• It utilises advanced syntax
• It is performed across multiple databases

How do I plan a comprehensive search?

To construct a comprehensive search, you must:

We suggest you create a concept table to capture all the required information for a comprehensive search. Download this template to keep track of your work.

Where do I search?

Each database provides access to different collections of resources, so you’ll need to think about which databases contain the information you’re looking for. If you don't search enough sources, you can miss out on crucial studies.

Depending on your review question, you need to consider searching both multidisciplinary and subject-specific databases.

• Multidisciplinary databases contain a high volume of literature from a diverse range of subjects. They are helpful in locating literature on your topic from disciplines you might not have considered.
• Subject-specific databases contain in-depth coverage of literature which is highly specific to a particular field.

A balance of both database types is required in your search strategy.

Cochrane considers Medline (via either Ovid or PubMed), Embase, and CENTRAL to be the most important sources of evidence to search when conducting a systematic review in biomedical disciplines.

Health and Medicine Databases

To learn about the main databases in Health and Medicine, download this document

Where not to search

Can I search OVID?
Ovid is a platform that hosts many databases including Medline, Embase, Cochrane Central, and PsycINFO. These databases are available on other platforms which may use different search syntax. When writing your protocol, make sure you mention the platform on which you will search the database, e.g. Medline via Ovid., CINAHL via Ebsco, etc.

Should I search PubMed or Medline or both?
The Medline database is available through a number of platforms, including paid (Ovid, Web of Knowledge etc.) and free (PubMed). PubMed is a free database maintained by the National Center for Biotechnology Information (NCBI) in the US. Whilst PubMed indexes some journals not included in Medline, most of the content in PubMed is available via the Library’s Ovid Medline subscription. Constructing your search on Ovid Medline provides increased control over search structure, aiding consistency in your methodology. When researchers don’t have access to Medline, they can search PubMed instead. PubMed may be particularly useful for animal model-based research questions and studies in genetics.

Should I search Google Scholar?
One of the important features of a systematic review search is its reproducibility. Because Google Scholar searches are not reproducible, we advise against including Google Scholar in the list of databases for comprehensive searching.
Google Scholar can be used for scoping searches, and you can use Google when searching for grey literature.

Grey literature

Grey literature refers to research material that hasn't been published commercially like a book or a journal article has. The challenge lies in locating it, as it can be either published or unpublished.

Examples of published grey literature include:

• conference papers
• theses and dissertations
• government and organisation reports
• clinical trial registers
• clinical guidelines

Examples of unpublished grey literature include:

• unpublished trials
• on-going studies
• unpublished data

Depending on your subject, it might be appropriate for you to include grey literature as evidence because it provides data which otherwise wouldn't be available through journal articles. Including grey literature also helps reduce publication bias.
We've compiled a list of tools and resources to assist you. You can also find useful tips on searching Google for grey literature in this document.

Search filters

Search filters are validated methods for finding relevant research in health databases. Filters are commonly used to limit searches to a particular study type, or to find articles on a specific subject. They can be used to enhance your search, limit your search results to only relevant study types, or for identifying additional search terms. It is not recommended to combine multiple study design filters – where your eligibility criteria includes multiple study designs, it is instead recommended not to use a study design filter.

Study type filters

Cochrane randomised controlled trial filters

McMaster University Hedges Project contains filters for the following databases:

• CINAHL
• Embase
• Medline
• PsycINFO

Search filters (Scottish Intercollegiate Guidelines Network)

PubMed Clinical Queries

Search filters (ISSG)

CINAHL Clinical Queries

Subject area filters

CareSearch Australia: find evidence-based literature via PubMed for palliative care and related topics

Integrated care: search integrated care literature via PubMed

Aboriginal and Torres Strait Islander Health: search for literature about ATSI populations and health issues via PubMed

Search Blocks (Biomedische Informatie): add 'blocks' to your database searches to limit them by topic

SYRCLE Filter for Animal Studies: designed to limit a search to relevant animal studies

Need help?

Academic Liaison Librarians can help Honours students, Higher Degree Research students, and staff plan search strategies. Don’t forget to bring your concept table to your appointment.

While we can suggest some resources to help you search for grey literature, actual searching is out of scope in our support for systematic reviews.

Additional Reading

• Aromataris, E., & Riitano, D. (2014). Systematic reviews. AJN, American Journal of Nursing, 114(5), 49–56.
• Belter, C. W. (2016). Citation analysis as a literature search method for systematic reviews. Journal of the Association for Information Science and Technology, 67(11), 2766–2777.
• Booth, A. (2016). Searching for qualitative research for inclusion in systematic reviews: a structured methodological review. Systematic Reviews, 5(1), 74.
• Bramer, W. M., De Jonge, G. B., Rethlefsen, M. L., Mast, F., & Kleijnen, J. (2018). A systematic approach to searching: an efficient and complete method to develop literature searches. Journal of the Medical Library Association, 106(4).
• Bramer, W.M., Rethlefsen, M.L., Kleijnen, J., & Franco, O.H. (2017). Optimal database combinations for literature searches in systematic reviews: a prospective exploratory study. Systematic Reviews, 6(1), 245.
• McGowan, J., Sampson, M., & Lefebvre, C. (2010). An evidence based checklist for the peer review of electronic search strategies (PRESS EBC). Evidence Based Library & Information Practice, 5(1), 149-154.
• McGowan, J., Sampson, M., Salzwedel, D. M., Cogo, E., Foerster, V., & Lefebvre, C. (2016). PRESS peer review of electronic search strategies: 2015 guideline statement. Journal of Clinical Epidemiology, 75, 40–46
• Paez, Arsenio. (2017) Grey literature: an important resource in systematic reviews. Journal of evidence-based medicine, 10(3), 233-240.

̏

Perform searches

Comprehensive searching is a complex skill that is refined through deliberate practice. This section offers tips on how to execute comprehensive searches effectively and efficiently across multiple databases.

Where can I get help with searching specific databases?

Before you start searching in a database, check the following:

• Does the database have a thesaurus? (look for a “map to subject headings/suggest subject terms” function on the database search page)
• What syntax does this database use? (look for search tips in the database Help section)
• Does this database have a Save Search function?

In a database with a thesaurus, you will need to search for subject terms as well as for keywords.

Syntax are search operators which can be used to modify and improve the results of keyword searches. They'll vary according to the database. See Flinders University's database syntax guide for a comprehensive table of operators used across different databases.

To learn more about specific databases in Medicine and Health, their subject headings, syntax, and search techniques, enrol in Ready for Health Research and complete Modules 3-4.

Below you can find quick database search guides for specific databases:

Database search guides

How can I run the same search in all selected databases?

Because databases use different syntax and subject headings, you'll need to adapt your strategy to fit each database you search. See Flinders University's guide to translating searches from Medline to other databases like CINAHL, PsycINFO, and Scopus.

On Ovid, you are NOT advised to:

• Re-run a search created on one database (e.g., Medline) in the other database (e.g., Embase) due to the differences in subject headings
• Use the Ovid function to exclude Medline content from Embase - this will interfere with your number of search results, recorded later in the Prisma flowchart (see below)
• Select multiple databases and search them simultaneously

What is the right number of search results for a systematic review?

There is no magic number of search results that a systematic review search should aim for. The number of retrieved results will vary depending on the topic and the area of study. Established fields will yield more results, while new fields will yield less. A comprehensive search strategy needs to balance two objectives: being inclusive enough to capture as many relevant results as possible while keeping irrelevant results to a minimum.

If your search returns too many results, you might need to:

• Be more selective with the subject terms you are using (i.e., reconsider applying the “explode” function)
• Add more main concepts
• Reconsider using common words as keywords

If your search returns too few results, you might need to:

• Include more synonyms in both subject and keyword searches
• Consider adding less specific subject headings (broader terms) or using the “explode” function
• Consider adding more keywords
• Reduce the number of main concepts

How do I check the relevance of my search results?

A good way to check if your search is working is by creating a small sample of relevant articles (5 to 10) and checking if your search retrieved these articles. You can quickly find the relevant articles for the sample set by doing a quick search in Google or PubMed.

Search for your topic in the PubMed PubReMiner tool to check why your search might not be working. Whereas a search in PubMed retrieves relevant articles, a similar search in PubReMiner retrieves metadata assigned to the articles during the indexing process, such as author names, journal titles, and subject headings, and calculates the frequency with which the same entries appear in the records. Identify high frequency subject headings used to index the relevant articles and include them in your Medline search. Alternatively, if the articles in the same topic are indexed with a wide range of low frequency subject headings, consider including keywords that consistently appear in titles and abstracts of the relevant articles.

When do I stop searching?

When conducting a systematic review, your aim is to search enough databases to be able to state with confidence that the literature has been comprehensively searched. Indicators of saturation include:

• You look in a different database and don’t retrieve any new papers
• You can identify no new data. Click on the Sage Knowledge logo to view content
• You use a rule to decide when to stop searching

How do I document my searches?

Most systematic reviews include a PRISMA flow diagram in their methods section, along with the search strategies themselves. Record your searches in a log or a document to refer to them when writing up your methodology. You can use the PRISMA chart generator to help you build a PRISMA flow diagram for your own systematic review.

To ensure accurate reporting, keep numbers of which records were obtained from each database. Set up folders in your reference manager to help keep track of this.

You can refer to the Cochrane Handbook for tips on which elements of your search need to be documented

Other methods of locating relevant studies for your review

To ensure that you’ve located all relevant studies, you may consider supplementing your comprehensive search with one or more of the following additional types of searching:

• Handsearching - review the reference lists of papers you have already located for more relevant studies, or choose to search specific journals which apply to your research area. Journals selected for handsearching should be identified from your database searching (Aromataris & Riitano, 2014; Booth, 2016).
• Contacting authors - contact authors of conference papers or relevant studies to clarify details of a study or request more data. They may also provide papers that weren't retrieved through your comprehensive search (Aromataris & Riitano, 2014; Booth, 2016).
• Citation chaining – identify a small number of key papers on your topic, and then find all the publications they cite and are cited by to produce a set of closely related publications.
• Grey literature – searching grey literature is imperative for capturing the unpublished studies. Refer to Plan Search and Select Databases section for more detail.

Any studies that were located by means other than a reproducible comprehensive database search (including grey literature) should be recorded in a separate box in your PRISMA diagram (Additional records identified through other sources).

Need help?

We recommend completing the Ready for Health Research Canvas course for more help performing database searches for a systematic review. You might benefit from completing the course even if you are not a health and medicine researcher. Honours students, Higher Degree Research students, and staff are recommended to complete the module before making an appointment with an Academic Liaison Librarian.

Additional Reading

• Higgins JPT, Thomas J, Chandler J, Cumpston M, Li T, Page MJ, & Welch VA (editors). (2019). Cochrane Handbook for Systematic Reviews of Interventions. Version 6.0. Cochrane. Available from www.training.cochrane.org/handbook.
• Gough, D., Oliver, S., & Thomas, J. (2017). An introduction to systematic reviews (2nd ed.). Los Angeles: SAGE.
• Liberati, A., Altman, D.G., Tetzlaff, J., Mulrow, C., Gøtzsche, P.C., Ioannidis, J.P.A., Clarke, M., Devereaux, P.J., Kleijnen, J., & Moher, D. (2009). The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate health care interventions: Explanation and elaboration. PLoS Medicine, 6(7), e1000100.

Screen Results

Comprehensive searches always retrieve more results than will be included in your final analysis. The screening process allows you to filter out irrelevant results by applying your inclusion and exclusion criteria. This section offers tips on efficient screening.

Which tool can be used for screening?

There are many collaborative tools for screening, both free and paid, to make the process easier. The Library doesn't currently support any screening tools.

Popular paid tools include:

Covidence - a tool for screening and data extraction. Recommended by Cochrane. Your first review can be conducted free of charge. Subscription fees apply for additional reviews.

Eppi-Reviewer - a comprehensive tool used for the searching, screening, and synthesis stages of a systematic review. Freely available to Cochrane-affiliated researchers.

Free alternatives include:

A Covidence free trial - trial allows 2 reviewers to conduct 1 review.

Rayyan QCRI - a web-based tool for collaborative screening. Learns selection patterns to assist with labelling of studies.

Abstrackr - a web-based tool for collaborative screening and labelling abstracts for relevance.

Picoportal - a systematic review platform.

More screening tools (both paid and free) can be found using the Systematic Review Toolbox - a comprehensive catalogue of tools to support systematic reviews.

How do I use EndNote for screening?

The Library supports EndNote which can be set up for managing search results for a systematic review.

Please refer to this subject guide to learn a 10-step process to use EndNote for systematic reviews.

How do I export searches from different databases?

Before you screen your search results, you will need to export them from each database. You should only start exporting when all your search strategies are finalised, and no further edits are anticipated.

You can find instructions on how to export from multiple databases on the EndNote subject guide (look under the Specific Database guides tab).

Refer to this table to learn EndNote export limits across different platforms and tips on bulk exporting.

How do I screen search results?

To control bias and minimise the risk of excluding relevant studies, screening is usually done by two or more independent reviewers. They talk through any disagreements in decision making over study inclusion or exclusion.

A standard workflow for this stage might be:

1. Remove duplicate records from your pool of results
2. Examine titles and abstracts to remove articles which obviously don't meet the inclusion criteria
3. Find the full text of all potentially relevant articles
4. Link together any reports which discuss the same study
5. Screen full text articles and assess for inclusion in the review against the criteria outlined in your protocol.
6. Make any final decisions on study inclusion
7. Record numbers of articles included and excluded at each step in the PRISMA Flowchart

If your resources are limited, you might choose to do the first screening yourself and involve the second reviewer for the second screening when selection criteria are applied more rigorously, and for the full-text analysis.

Where can I get full text articles?

If using EndNote, you can attach most full text articles automatically using the ‘Find Full Text’ function. For the rest, check Library Search and, when you have found the full text, attach it to your article records in EndNote manually (refer to the EndNote manual for instructions). If you are confident that the library does not have access to the required article, you can request it via Resource Sharing.

How can I re-run the search?

Many publishers will request that you re-run your search if six months or more lapsed between the last search and the date of review admission for publication.

To do so, you can run your saved searches and limit them to the last year of search, and then screen results just for the last year to identify any new relevant articles, and update your PRISMA flowchart accordingly. Refer to this document for instructions on how to use EndNote to manage references when re-running systematic review searches.

Need help?

Librarians cannot help you with screening; for assistance, contact your supervisor or a subject expert in your faculty.

Additional Reading

• Gough, D., Oliver, S., & Thomas, J. (2017). An introduction to systematic reviews (2nd ed.). Los Angeles: SAGE.
• Lefebvre C, Glanville J, Briscoe S, Littlewood A, Marshall C, Metzendorf M-I, …Wieland LS. (2019). Chapter 4.6: Searching for and selecting studies.. In: Higgins JPT, Thomas J, Chandler J, Cumpston M, Li T, Page MJ, Welch VA (editors), Cochrane Handbook for Systematic Reviews of Interventions version 6.0. Cochrane.
• Li T, Higgins JPT, Deeks JJ (editors).(2019). Chapter 5: Collecting data. In: Higgins JPT, Thomas J, Chandler J, Cumpston M, Li T, Page MJ, Welch VA (editors). Cochrane Handbook for Systematic Reviews of Interventions version 6.0. Cochrane.
• Moher, D., Liberati, A., Tetzlaff, J., Altman, D.G., & the PRISMA Group. (2009). Preferred reporting items for systematic reviews and meta-analyses: The PRISMA statement. PLoS Medicine, 6(7), e1000097.
• Shamseer, L., Moher, D., Clarke, M., Ghersi, D., Liberati, A., Petticrew, M., Shekelle, P., Stewart, L.A., & the PRISMA-P Group. (2015). Preferred reporting items for systematic review and meta-analysis protocols (PRISMA-P) 2015: Elaboration and explanation. BMJ, 349, g7647.
• Verhagen, A.P. (2017). The art of systematic reviews. Musculoskeletal Science and Practice, 31, iv-vi.

Data extraction and appraisal

Data appraisal is essential for a systematic review. This section provides tips and suggests tools for efficient data extraction and appraisal.

How do I extract data?

Your next step is extracting the necessary data from your included studies. Not only will this help you assess the validity of the studies you're including in your review but will also help you prepare for the next stage: analysis and interpretation.

Before you begin to extract data from the studies, you need to decide what data elements are required for your analysis. The below table is a sample of items to consider. The full table can be viewed in the Cochrane Handbook.

Once you've decided which items are needed, use a data extraction form as a consistent way to record the information.

How do I appraise selected studies?

When appraising each study eligible for inclusion in your review, you'll be evaluating whether you can draw conclusions about the effect of the intervention.

You can assess this by examining the studies for various types of bias, which can lead to the study overstating or understating the true effect of the intervention. Section 8 of the Cochrane handbook names the following forms of bias:

• Selection bias
• Performance bias
• Detection bias
• Attrition bias
• Reporting bias
• Other biases

Appraisal tools

There are a number of appraisal tools which will help you assess the publications / studies you're including in your review.

How do I avoid incomplete reporting?

Incomplete reporting can affect how you assess risk of bias. What's excluded has a bearing on whether you can trust the study as a whole, or if you can only trust specific outcomes. Some ways of detecting whether a study has been reported incompletely include:

• Using study appraisal checklists
• Comparing the methods to the results
• Comparing the outcomes of a protocol (if there’s one available) to the published article
• Asking the study authors for more information

After appraisal, you might find there are further studies to exclude from the review based on your eligibility criteria. Make sure you record the reasoning behind exclusion as it will need to be reported.

Need help?

It's a good idea to talk to your supervisor or a statistician from the Sydney Informatics Hub about how you plan to conduct your analysis, ideally before you begin data extraction. This ensures you're collecting relevant data to support the analysis you plan to undertake.

Additional Reading

Cochrane has a GRADE Approach to rating the certainty of evidence in systematic reviews and creating a ‘Summary of findings’ table.

• Aromataris, E., Munn, Z. (Editors). (2020). JBI Manual for Evidence Synthesis. JBI. Retrieved from https://synthesismanual.jbi.global.
• Buccheri, R.K., & Sharifi, C. (2017). Critical appraisal tools and reporting guidelines for evidence-based practice. Worldviews on Evidence-Based Nursing, 14(6), 463-472.
• Centre for Evidence Based Medicine. (2018). Critical appraisal tools. Retrieved from https://www.cebm.net/2014/06/critical-appraisal/.
• Greenhalgh, T. (2010). How to read a paper: The basics of evidence-based medicine. Hoboken, NJ: Wiley-Blackwell.
• Higgins JPT, Savović J, Page MJ, Elbers RG, Sterne JAC. (2019). Chapter 8: Assessing risk of bias in a randomized trial. In: Higgins JPT, Thomas J, Chandler J, Cumpston M, Li T, Page MJ, Welch VA (editors). Cochrane Handbook for Systematic Reviews of Interventions version 6.0. Cochrane.
• Lefebvre C, Glanville J, Briscoe S, Littlewood A, Marshall C, Metzendorf M-I, Noel-Storr A, Rader T, Shokraneh F, Thomas J, Wieland LS. (2021). Chapter 4: Searching for and selecting studies. In: Higgins JPT, Thomas J, Chandler J, Cumpston M, Li T, Page MJ, Welch VA (editors). Cochrane Handbook for Systematic Reviews of Interventions version 6.2. Cochrane.
• Li T, Higgins JPT, Deeks JJ (editors).(2019). Chapter 5: Collecting data.. In: Higgins JPT, Thomas J, Chandler J, Cumpston M, Li T, Page MJ, Welch VA (editors). Cochrane Handbook for Systematic Reviews of Interventions version 6.0. Cochrane.
• Shea, B.J., Reeves, B.C., Wells, G., Thuku, M., Hamel, C., Moran, J., Moher, D., Tugwell, P., Welch, V., Kristjansson, E. & Henry, D.A. (2017). AMSTAR 2: a critical appraisal tool for systematic reviews that include randomised or non-randomised studies of healthcare interventions, or both. BMJ, 358, j4008.
• Taylor, P., Hussain, J.A., & Gadoud, A. (2013). How to appraise a systematic review. British Journal of Hospital Medicine, 74(6), 331-334.
• The National Academies of Sciences, Engineering, and Medicine. (2011). Standards for systematic reviews. Retrieved from www.nationalacademies.org/hmd/Reports/2011/Finding-What-Works-in-Health-Care-Standards-for-Systematic-Reviews/Standards.aspx.
• University of South Australia. (n.d.). Critical appraisal tools. Retrieved from http://www.unisa.edu.au/Research/Health-Research/Research/Allied-Health-Evidence/Resources/CAT/.
• Young, J.M. & Solomon, M.J. (2009). How to critically appraise an article. Nature Reviews: Gastroenterology & Hepatology, 6, 82-91.

Analyse and interpret

To answer your research question, you will be analysing and synthesising the extracted data. This section provides tips and suggests tools for efficient data analysis.

Your review will lend itself to different methods of analysis:

• A narrative analysis involves preparing a structured summary, comparing the data and coming up with themes to find evidence of an effect.
• A statistical analysis is also known as a meta-analysis. It involves pooling and analysing the results across a number of studies using statistical methods.
• An integrative analysis combines both narrative and statistical analysis techniques, to reflect the diversity of the evidence included. For example, an integrative analysis could be a meta-analysis which appears alongside a narrative synthesis which provides additional information about the effect.

Narrative synthesis techniques

Why should I conduct a meta-analysis as part of my systematic review?

1. To increase power - the chance of detecting a real effect as statistically significant (if it exists)
2. To improve precision
3. To answer questions not posed by the individual studies
4. To settle controversies arising from apparently conflicting studies
5. Generate new hypotheses

Cochrane Handbook, Section 10.2 'Introduction to meta-analysis'

Despite these advantages, a meta-analysis should only be performed if you have a group of studies which are similar - this similarity is referred to as homogeneity. At the very least, studies should be homogeneous in terms of participants, interventions, and outcomes so the meta-analysis can draw a meaningful conclusion.

Any kind of variability among studies is referred to as heterogeneity. There are several kinds - Cochrane names:

• Clinical diversity - where participants, interventions and outcomes differ between studies
• Methodological diversity - where there is variability in study design
• Statistical heterogeneity - where the observed intervention effect differs between studies because of chance

It is important to measure and address heterogeneity in your review because it affects the extent to which you can rely on the conclusions you draw.

Software

NVivo
Qualitative data analysis software which enables you to handle rich text based information. It automates many manual tasks associated with analysis, like data classification and sorting. Available to University staff through ICT.

Need help?

Your supervisor or a subject expert in your faculty can assist with analysis and interpretation of your results.

If you're an HDR student, there is free statistical advice available through the Sydney Informatics Hub. Book an appointment to talk to a statistician by using the assistance request form.

Additional Reading

• Akobeng, A.K. (2005). Understanding systematic reviews and meta-analysis. Archives of Disease in Childhood, 90(8), 845-848.
• Booth, A., Papaioannou, D., & Sutton, A. (2012). Systematic approaches to a successful literature review. London: Sage.
• Deeks, J. J., Higgins, J. P., & Altman, D. G. (2011). Chapter 10: Analysing data and undertaking meta-analyses. In: Higgins JPT, Thomas J, Chandler J, Cumpston M, Li T, Page MJ, & Welch VA (editors).(2019). Cochrane Handbook for Systematic Reviews of Interventions version 6.0. Cochrane.
• Foster, M., & Jewell, S. (2017). Assembling the pieces of a systematic review: a guide for librarians. Lanham, Maryland: Rowman & Littlefield.
• Impellizzeri, F.M., & Bizzini, M. (2012). Systematic review and meta-analysis: A primer. International Journal of Sports Physical Therapy, 7(5), 493-503.
• Lee, C.H., Cook, S., Lee, J.S., & Han, B. (2016). Comparison of two meta-analysis methods: Inverse-variance-weighted average and weighted sum of z-score. Genomics & Informatics, 14(4), 173-180.
• Rice, K., Higgins, J.P.T., & Lumley, T. (2017). A re-evaluation of fixed effect(s) meta-analysis. Journal of the Royal Statistical Society, 181(1), 205-227.
• Riley, R.D., Ensor, J., Jackson, D., & Burke, D.L. (2017). Deriving percentage study weights in multi-parameter meta-analysis models: With application to meta-regression, network meta-analysis and one-stage individual participant data models. Statistical Methods in Medical Research, 27(10), 2885–2905.
• Uman, L.S. (2011). Information management for the busy practitioner: Systematic reviews and meta-analyses. Journal of the Canadian Academy of Child and Adolescent Psychology, 20(1), 57-59.

Write and publish

This section provides tips and considerations for reporting the findings of your systematic review and deciding where to publish.

How should I write up my review?

In your write-up, you are not only reporting your findings and answering your review question – you are also reporting on the decisions made and the procedures followed that led to your findings. To ensure that your write-up is complete and transparent, use the PRISMA checklist.

Where should I publish my review?

Where you publish has a huge effect on the visibility and impact of your systematic review. Here are some tips for finding the right journals to publish in according to the scope of your review and your target audience:

• Look at your reference list. Which journals do you regularly and consistently cite? Your work will likely be of interest to both the journal and its readers
• Make sure you check the aims and scope in the 'information for authors' pages of each journal you'd like to publish in (e.g. Journal of Clinical Nursing)
• Make sure the journal’s instructions to authors include enough detail and minimum standards for conducting and reporting reviews
• Journal suggester tools generate lists of suggested journals based on the similarity of your abstract to articles these journals have published. Be mindful of the coverage limits of suggester tools provided by journal publishers
• Evaluate journals using metrics to identify those most influential in your field. There are a variety of metrics you can use to inform your decision. Some are more useful for certain disciplines than others. Check the Strategic Publishing Toolkit for more information

How should I share my research?

When you submit your article to a journal for publishing, you often sign over copyright to the publishing journal. This affects how you can share it in the future, and how readers can access it. If it is important for your research to be accessible to clinicians and practitioners, check what sharing options a journal offers (often stated in open access or editorial policies on the journal/publisher website), or investigate an open access publication.

• For more information about copyright, check the Copyright pages on the Library website
• Open Access allows everyone to access your research. There are different ways of making your work Open Access. For more information, visit the Open Access pages on the Library website

Need help?

For assistance with writing, contact your supervisor.

In addition to providing online modules on writing, The Learning Hub also runs writing workshops.

If you have a question about copyright, contact the Copyright Team.

Enquiries about Open Access publishing can be directed to an Academic Liaison Librarian. If you have a question about depositing a copy of your publication into the University repository, contact the Repository Team.

The Library runs workshops on Strategic Publishing. Check the Library Calendar to see upcoming workshops and to register.

Additional Reading

• Haddaway, N. R., Bethel, A., Dicks, L. V., Koricheva, J., Macura, B., Petrokofsky, G., ... Stewart, G. B. (2020). Eight problems with literature reviews and how to fix them. Nature Ecology & Evolution, 4(12), 1582-1589.

What is the most effective way to search a database?

What search strategies does the Library recommend using on the databases?

What other techniques can I use to develop effective search methods?

What other strategies might you use beside key word searches of databases to find research articles for your literature review?