Systematic and Scoping Review Workshop Series: Workshop 3: Streamlining your review with Covidence and Zotero

Please download the practice files (below); one is a text file exported from Pubmed ( Pubmed export), and the other file is a RIS file exported from Web of Science. These are the data files that you will be able to import into Covidence and Zotero during the session. 

Database: Pubmed

Search strategy (example):

(((("communication technolog*") OR ("Information Technology"[Mesh] OR "Digital Technology"[Mesh])) AND (("Aged"[Mesh]) OR ((elderly[Title/Abstract] OR older[Title/Abstract] OR senior[Title/Abstract]) AND citizen*[Title/Abstract] OR person*[Title/Abstract] OR people[Title/Abstract]))) AND (("nursing home"[Title/Abstract] OR "care home"[Title/Abstract] OR "retirement home"[Title/Abstract] OR "assisted living"[Title/Abstract] OR "long*term care"[Title/Abstract]) OR (("Long-Term Care"[Mesh]) OR "Residential Facilities"[Mesh]))) AND (("Social Isolation"[Mesh]) OR ("social isolat*"[Title/Abstract] OR "social loneliness"[Title/Abstract] OR lonel*[Title/Abstract]))

Database: Web of Science (WoS)

Search strategy (example):

digital OR communicat* OR information* NEAR/3 (technolog* OR device*) (Topic) and aged OR senior OR older OR elderly NEAR/3(person* OR people* OR citizen* OR person*) (Topic) and "long?term?care" OR "nursing home" OR "retirement home"(Topic) and loneliness OR isolation (Topic)

Zotero is an open-access, easy-to-use reference management tool that helps you collect, organize, cite, and share your research sources.  

Zotero allows you to save references from library catalogs, research databases, and the Web; upload and organize PDFs, images, audio and video files, snapshots of web pages, and more; write annotations and attach them to citations; and create bibliographies using most major citation styles.

Send yourself an invitation and register for a Covidence account

Please make sure to use your uWaterloo email account when registering for Covidence; this will ensure that you can access the University's institutional licence, which access to unlimited reviews. Covidence reviews can also be shared with colleagues outside of the University Waterloo community. 

What is Covidence?

Covidence is a web-based platform, and no additional software installation is required.There is no desktop version of Covidence, it is web-based only. Covidence acts as a digital tool that researchers can use to streamline the screening and data collection of comprehensive literature review. And because reviews are often undertaken in teams, not only to minimize bias but also just to share the volume of work, Covidence excels at being a collaborative hub for research teams. Teams can ensure consistency for their roles in the review, as well as for data collection and extraction.

Covidence is very specific to the workflow of a review article, and it is currently for screening and reviewing journals articles only. Covidence does not take the place of other important tools that researchers may need during the compilation of a review, such as a reference manager (Zotero or Endnote) or the statistical software necessary for meta-analysis such as RevMan Web (Cochrane). Covidence works in conjunction with these tools and researchers can import and export data across tools. The Cochrane Library recently recognized Cochrane as the primary screening and data extraction tool for Cochrane authors conducting standard intervention reviews. The quality assessment template in Covidence 2.0 and 1.0 is based on The Cochrane Library's Risk of Bias (RoB) template. 

Figure 1: A simplified structured review process, from Planning your review (with materials and methods covered in workshop 1 of this series); and searching for evidence strategically and systematically  (workshop 2); and then organizing, screening and appraising evidence; followed by writing your review and publishing your results. Workshop 3 will cover technical aspects of the process for organizing, screening and appraising evidence (Covidence and Zotero) stage, as well as a brief preview of the writing with your references stage (Zotero). 

Review settings

Remember when we created our Practice Review? Now we are returning to some of our initial selections of Review Type and Title, as well as many additional options. Review Settings are the first screen in the settings menu. This is where a researcher can re(name) their review; choose their review type, such as an umbrella review, scoping review, narrative review, and more. Researchers can also add the search strategy; this can be very useful to have updated for large team reviews. 

In this screen you can also change the number of reviewers from 2 to 1. The default is for two reviews for the purpose of minimizing bias. However, when utilizing Covidence’s Extraction 2.0, a researcher can change the number of reviewers from two to one which can be helpful if you are using this tool on your own. 

The default setting  of two reviewers is mandatory for Covidence’s Extraction 1.0 tool, which is designed specifically for interventional Cochrane reviews. For the sake of minimizing bias I would highly recommend having two reviewers for any structured review, including scoping and literature reviews. 

For our practice review let's add our search strategy details:

• Date of Last Search
• Strategy: 

Database, Pubmed

(((("communication technolog*") OR ("Information Technology"[Mesh] OR "Digital Technology"[Mesh])) AND (("Aged"[Mesh]) OR ((elderly[Title/Abstract] OR older[Title/Abstract] OR senior[Title/Abstract]) AND citizen*[Title/Abstract] OR person*[Title/Abstract] OR people[Title/Abstract]))) AND (("nursing home"[Title/Abstract] OR "care home"[Title/Abstract] OR "retirement home"[Title/Abstract] OR "assisted living"[Title/Abstract] OR "long*term care"[Title/Abstract]) OR (("Long-Term Care"[Mesh]) OR "Residential Facilities"[Mesh]))) AND (("Social Isolation"[Mesh]) OR ("social isolat*"[Title/Abstract] OR "social loneliness"[Title/Abstract] OR lonel*[Title/Abstract]))

• Now for the purposes of working through the references for our Practice Review, let's change all of the Reviewers Required to 1.
• Ensure that extraction 2 is selected and SAVE.

Covidence support

How to configure settings, article and video

Learn more about using Covidence for a Cochrane Review, article and video

Team Settings

In the Team Settings screen, researchers can check in on the teams' progress, as well as set the screening and review rules for the team. There are currently no team settings' options for Data Extraction. 

The default option is Everyone can do everything this means there are no rules as to who can screen, review, extract data and decide consensus.  When working on very large reviews, or perhaps multi-site reviews this can be a very useful screen for assigning a group to screen for example and another groups solely focused on extraction. The data extraction stage can take quite awhile and therefore if you have references that meet all you inclusion criteria for their abstract, and then for the full-text, having roles assigned to review members can be helpful for managing time spent on the review. 

Select Manage rules in order to determine who on the team does what during the review process. 

The All studies must be screened by option lets you define who must screen each study in the stage. You could use this functionality to ensure that an experienced reviewer(s) will at the very least screen all studies.

The Conflicts can be resolved by option lets you control who can resolve conflicts during the screening process.  Any reviewer who is a member of this group will have the permission to resolve conflicts.  If no one is assigned to the group then everyone will be able to resolve conflicts. 

Covidence support

How to oversee a review and setup who does what, video and article.

Study Tags

Customized tags offer a convenient way to quickly add info and keep track of studies. The default tags for studies ongoing, and studies awaiting classification can not be deleted as these tags are counted for the PRISMA chart. 

Covidence support

How to create and manage study tags, article.

Eligibility criteria 

Covidence reworked this section for a major software update in July 2023. The PICO structure is now explicit and you can use it to organize your inclusion and exclusion criteria. However, if you aren't using a PICO structure or aren't sure where to enter a term, you can always use the Other Inclusion and Exclusion field.

• Utilizing our practice review, let's enter our PICO criteria; one-term per line and you can edit later.

• SAVE ! and scroll back up to the top of the eligibility screen to now manage your Highlights and Full-text Exclusion reasons. 

Highlights

Covidence allows you to input certain keywords that might indicate inclusion or exclusion for reviewers. Inclusion key words or phrases will be highlighted in green, and exclusion key words or phrases will be highlighted in red. This section is incredibly useful for larger teams as all of the key words and phrases can be customized for your review. Note that the criteria for highlights is one key word or phrase per line, and that Covidence only finds EXACT matches; spelling variations will not be captured.

• Let's ADD+ some highlights for our practice review:

These exclusion criteria are going to be more specific as they will be used for the full-text review. Covidence pre-populates the list of exclusion reasons with popular options, such as wrong does, paediatric population, wrong setting, wrong study design, wrong intervention etc. 

If you like, you can add your own exclusion reasons as well as delete those that there by default. 

Inclusion reasons are not used in full-text review; you have already indicated in title and abstract screening that you are including the article for further review of the full-text, so the full-text review is just going to be a matter of excluding the article from the final data extraction.

• Always SAVE before leaving the settings page.

Full-text exclusion reasons

Covidence Support

Creating and managing eligibility criteria, article and video

Importing to Zotero 

We are going to great a new shared group library in Zotero. In Zotero we can organize our references, attach full-text, attach annotations and also share our libraries with a group. 

1. Open the desktop version of Zotero, and minimize to the background
2. All shared group libraries must be created online first. Go to zotero online and sign-in with your registered login. 
3. In Zotero online:
   1. In the top menu, choose GROUPS, and then select Create a New Group.
   2. Name the group: practice_review_yourname, and choose the membership value. Let's do Public closed membership, meaning that you have to be invited to participate in the group, but zotero users can also find this group and request access. 
   3. For Group Settings, let's confirm our settings. Note that open, public Groups can't share stored files (pdfs) due to copyright, so this is why it is important to have a closed or private group. Save Settings
4. In Zotero desktop:
   1. Maximize your desktop version of Zotero, and click sync (top right, green arrow). 
   2. You should now see your shared, Practice_Review group library in the Group Libraries of your Zotero desktop. If you see your new group library, you can close the browser window for Zotero online.
   3. In your Zotero desktop, File > Import our two practice files - the exact same files you used to import into Covidence. Make sure that your desktop is open to the Group Library, Practice_Review.
   4. Once both files are imported, select all and right-click to bring up a menu -- select Fine available PDFs. Zotero will use open access links as well as the proxy with the University of Waterloo to try and match each reference to the full-text.
5. Minimize and return to Covidence for the time-being.

A "Maybe" vote is treated the same way as a "Yes" vote, meaning that it will move that citation forward in your review. 

If in dual reviewer mode, you can filter those citations that you have cast one "Maybe" vote on by using the filter function at the top of your 'Awaiting other reviewer' list. If in dual reviewer mode and a citation receives two "Yes"/"Maybe"votes it will move forward in your review and you'll be able to see those votes cast by clicking on "History" underneath each citation.

If the second vote is a "No" then that citation will move to your 'Resolve conflicts' list. Here you won't be able to see those "Maybe" votes unless they were cast by you, as seeing your co-reviewer's previous votes can lend itself to biasing the adjudicating vote. 

You won't be able to generate your list of citations that received a "Maybe" vote during title and abstract screening. All voting is blinded meaning that if you are in dual reviewer mode, your co-reviewer won't be able to see which citations you voted "maybe" on until they have cast their vote, and if this vote is also a "Yes" or a "Maybe".  

Action items:

de Andrade, 2017

1. In Covidence, find de Andrade 2017
   1. Is the full-text missing? If the full-text is missing, this just means that Covidence was not able to locate an open access version of the full-text for this article. 
2. In Zotero, find de Andrade, 2017
   1. Zotero has found the full-text through the University of Waterloo proxy. Now I could manually upload this pdf, but instead I am going to wait and do a bulk import of pdfs because I am sure that I will have other full-text missing. 
3. To see what references still need full-text, in Full-text review select the 'Needs Full Text Filter' to see al the full-text that we are missing.

Delello, 2017: 

1. In Covidence, find Delello 2017.
   1. The full-text is missing so let's look in Zotero
2. In Zotero, find Delello 2017
   1. The proxy has not located the full-text, which means that we have to go and find this full text or order it in from interlibrary loan.
   2. Copy the DOI (digital object identifier) for this record
   3. Paste the DOI in OMNI, which is the library catalogue for the University of Waterloo. 
   4. Download and save the pdf from OMNI
   5. Manually attach this pdf to the record in Zotero and proceed to bulk import. 

Bulk import of missing full text 

So there's a little bit of back and forth for this function, but once you get the hang of it this is an incredibly useful tool. For this function, we are going to be working with xml files in Covidence and Zotero. 

For our practice review, we have actually already done the first two steps in the batch import, because we have utilized the same data files (from Pubmed and Web of Science), for import into both Covidence and Zotero.  As a result, we are going to go straight to Zotero and generate an xml of our  practice review references. 

1. Go to you zotero desktop. Select all references in your practice review file, and right click to generate a menu. Select 'Export Items' from the menu.
2. Export Items to Endnote XML, ensure all options are checked, and call the file: practice_review_batch1. Save. Zotero will create a folder for your xml file, and the attached pdfs. 
3. Return to Covidence in your browser and navigate to full-text review/ bulk import of missing full text. 
4. Jump right to Step 3 of the bulk import function, and choose the xml file, practice_review_batch. Upload.
5. Covidence will scan the xml to match any missing full text attachments. 
6. If full-text pathways are found, Covidence will let us know how many we have to upload. 
7. Proceed to upload pdfs by selecting the PDF folder in the Zotero generated-folder (see 2) and clicking import on the whole folder. Covidence should upload and match all of the pdfs to their bibliographic records.
8. Return to full-text review and apply the 'Missing full Text' feature -- how many do we have now? 

For large reviews it is normal to do multiple batch imports as articles on order from archives or interlibrary loan can come through sporadically during the screening and review process. If you can't find an article or if you are continually running into paywalls, ensure that you are utilizing the University proxy and ask us at the library for assistance anytime :) 

Covidence Support

Learn more about bulk upload: use your reference manager (e.g. Zotero, or Endnote) to bulk upload full text.

Data Extraction template

The data extraction templates is pre-populated with generic options for data extraction, such as study ID, author details, methods, baseline population characteristics, interventions and comparisons, outcomes, etc. These fields are going to be specific to your team's review, so this template can be fully customized. 

This is also where you can publish instructions for reviewers, so that everyone is on the same page. You must publish the template in order to utilize your extraction fields. 

Covidence support

• How to create, publish and edit a data extraction template, video and article.
• How to export data to excel using extraction 2.0, video and article
• What does extraction 2.0 data look like when exported to Excel? video and article
• How to use multi-select checkboxes, article

Quality Assessment template 

The default QA template is Cochrane’s Risk of Bias. If you want to use a different tool, select I want to start from scratch in the Editor to delete the RoB domains and start with an empty template.

If you don’t want to do quality assessment in your review, don’t publish your QA template and QA will not appear for the extractors of your review. 

Covidence support

How to start a quality assessment, video and articles

Action items:

Based on our work in the title and abstract screening (step 2), and in the review of the full-text (step 5), we should have at least 2 studies in data extraction: Cotten 2013, and Delello 2017. 

In Covidence, return to the main screen for our practice review, and scroll down to the last category, Extraction. Covidence will show you what articles  are ready for data extraction, and to avoid reviewers working on the same extraction, your progress for the extraction will always be saved.  Extraction can also take quite a bit of time, so being able to save and pause your progress with reviews in this category is an incredibly useful function of Covidence. 

For this data extraction, we are going to focus on Cotten (2013). But before we can extract any data, we have to publish a data extraction template. 

1. Go to data extraction template in the top right corner of you Covidence window. 
2. The draft data extraction template is the default, and it is a template that is really streamlined for health inventional studies, so fairly specific. You can extract anything you like for your research though, and so the template is completely customizable; choose "I want to start from scratch". But for now, let's add a feature and some instruction to the default data extraction template as this fits our practice review quite well. 
3. Add + in single selection for Allocation or Concealment:
   1. Option 1: Random
   2. Option 2: Nonrandom
4. Add + instructions for your extractors on any feature.
5. Publish the data extraction template
6. Return to the extraction stage. Now extraction can begin and the Data extraction template will be the standard for everyone on your team.
7. Spend a few moments extracting some data from Cotten (2013). 
8. Compete your extraction
9. Confirm all of the entered data in the double-check, which is an incredibly useful tool for supervisors of larger reviews. 
10. Export your data to see the generated table. 

Exporting references

Data can be exported from Covidence in a variety of formats. References or bibliographic records can be exported at any point from Covidence in CSV format, Cochrane Registry format, or formatted for reference managers like Zotero, Endnote and Mendeley. 

Inter-rater reliability data for screening and full-text review can also be exported in CSV format. Inter-rater reliability refers to the degree to which different raters or observers produce similar or consistent results when evaluating the same thing. 

The extracted data, compiled by reviewers using the inclusion and exclusion criteria, can be exported in CSV format. The extracted data can be concsenus-only data for quality assessment or individual data. This is a critical component of Covidence 1.0 extraction for interventional studies. 

Covidence support, articles and videos

• How to export study lists to your reference manager, video and article
• How to export extracted data to Excel using Extraction 2, video and article
• What is inter-rater reliability? article
• How to export inter-rater reliability data, video and article

Writing with Zotero

Zotero instantly creates references and bibliographies for any text editor, and directly inside Word and Google Docs. With support for over 10,000 citation styles, you can format your work to match any style guide or publication.

1. Open your zotero desktop, and minimize
2. Open Microsoft Word and create a new blank document. 
3. Zotero automatically installs the extension for Microsoft word - check your top toolbar for the Zotero menu. 
4. Now let's work with a few of our references (workshop demo).

Zotero Support

Plugins for Zotero. Zotero

Zotero for mobile, and the barcode scanner. Zotero