Background
The Pandemic has changed life so completely in many parts of our lives that we are left to wonder “is this the new normal or a blip?” So it is with systematic reviews (SR). With the Covid restrictions making primary research problematic, the number of SR being undertaken has skyrocketed. Once the purview of well-resourced teams of experienced health professionals, SR are now being undertaken by PhD students with either a kindly supervisor or a fellow student looking for a quid pro quo acting as second reviewer. Indeed, some post graduate students are tasked with performing a systematic review solo, as a learning exercise. Disciplines such as Education, Business, Economics and Engineering are routinely producing systematic reviews and often enlisting the aid of those information specialists best equipped to guide them through the process: the health librarians. With librarian contact hours at a premium, how comfortable would we, as health librarians, feel directing these interested but very green potential reviewers to Cochrane’s Interactive Learning modules? Is this a job well done, perhaps the new normal? We thought we should investigate.
Objectives
Our team of six academic librarians, one a Cochrane author, with a combined 100+ years of experience in the athenaeumic arts, are seeking to determine if the eleven Cochrane Interactive Learning (CIL) modules (https://training.cochrane.org/interactivelearning), with an assumed fifteen contact hours, are sufficient to lead a novice through the systematic review labyrinth to achieve the goal of publication or presentation. More precisely: Can the Cochrane Interactive Learning modules take a willing student, with the merest scrap of a clinical question, and who knows that a systematic review is a thing but knows nothing of the required steps, from this position to publication or presentation? In achieving this goal there is also the requirement, or at least the very strong desire, that the participant’s passion for the subject remains undiminished and that their mental health is not impacted during the process. Increasingly the role of the information specialist is not merely to assist in choice of databases and refinement of search strategy, but to add reassurance that the path being travelled is the right one. We often answer questions or extract questions from reviewers who may be reticent to ask their supervisors. Can these modules do this? It is far too much to ask that any set of prepared teaching tools can answer the specifics of any given SR topic no matter how sound the pedagogy or epistemology, but can they administer surety in the process? We will investigate.
Review methods
A scoping search was performed to see if a systematic review of the literature concerning Cochrane Interactive Learning modules (CIL) was possible. Our question: Are Cochrane Interactive Learning modules helpful for a first-time systematic reviewer aiming to achieve a meaningful systematic review, and lead a healthy and relatively stress-free life during that time?
Our PICO describing this:
|
Population |
Novice systematic reviewers about to embark on a systematic review |
|
Intervention |
Cochrane Interactive Learning modules (CIL) |
|
Comparison |
Other available means of guidance through the systematic review process |
|
Outcomes |
Publishable systematic review without undue stress |
We selected the following key health databases to investigate: PubMed, EMBASE, CINAHL, PsycInfo and the Cochrane Library. “Cochrane Interactive Learning” was the essential search term for our specific search topic and formed our search strategy for all sources. Search results for CIL produced 0 in PubMed, EMBASE and PsycInfo, with 1 in the Cochrane Library and 2 in CINAHL.
Clearly there is a gap in the published literature and insufficient evidence to perform a systematic review on the capacity of CIL to support creation of a systematic review. Running our eyes down the evidence pyramid with our desire attuned to creating the strongest piece of evidence possible, (and given available time and resources), we land at the bottom of the pyramid with expert opinion:
While the term “expert” is a daunting one and one that few librarians wear or tout comfortably, we could without reservation put forward our “experienced” opinion. To inform this, our team of six health librarians undertook a random, non-blinded, mixed methods approach to reviewing the Cochrane Interactive Learning modules https://training.cochrane.org/interactivelearning, courtesy of a 30 day free trial.
Data collection
Overview of Cochrane Interactive Learning modules
https://training.cochrane.org/interactivelearning
Information about each module and their learning outcomes is provided on the Cochrane Interactive Learning webpage, including the following brief description:
Developed by world-leading experts, this course provides over 15 hours of self-directed learning on conducting a complete systematic review process for both new and experienced review authors.
A short video is also provided, which recommends Cochrane Interactive Learning “to gain a more in-depth understanding of the process of conducting a systematic review. Trainers may use it for blended learning, combining the online modules as an introduction to more advanced learning in a face-to-face setting.”
There are a total of eleven modules (with the first module free to access):
The modules are designed as “Learning resources on key areas of Cochrane review methodology” (https://training.cochrane.org/cochrane-methodology). They provide links to relevant sections of the Cochrane Handbook and other resources to support producing a systematic review, including the following - https://training.cochrane.org/resource/good-practice-resources-new-cochrane-authors
Cochrane provides information about access, including accessibility and purchasing considerations:
Evaluation of modules using the CRAAP test
https://youtu.be/kMWcxhs8_F0
Currency:
While much of the information is timeless, some of the interfaces demonstrating processes need revision. Last updated in December 2019, these modules indicate a planned review date for Quarter 4, 2020. There is no indication of the reason for this delay, however this plan may have been impacted by COVID-19, like most things in this post-pandemic world.
Relevance:
The modules are primarily intended for Cochrane authors undertaking an intervention review. There is some content aimed at Cochrane authors only. Non-Cochrane authors, particularly those with limited systematic review experience, will find the first module, “Introduction to conducting systematic reviews”, provides a good overview of the systematic review process. In and of themselves, these modules are not stand-alone and require the user to explore the linked resources in order to get an in-depth understanding of the full systematic review process. Module 3, “Searching for studies”, has insufficient detail to enable the beginner to complete a systematic search, however it does provide the advice to seek assistance from an information specialist.
Authority:
Cochrane reviews are generally considered the gold standard and the interviewees featured in the modules’ videos are experts in their fields.
Accuracy:
The content of the modules is accurate, with tone and language suitable for the intended audience, apart from the previously mentioned currency issues, e.g. the need to incorporate changes in some online interfaces. Each section links to supporting material, particularly the specific sections in the Cochrane Handbook and other recognised authorities, although wandering off into these resources will significantly extend the estimated completion time for the modules.
Purpose:
The purpose of CIL is to educate Cochrane authors. It provides one of several Cochrane training methods to inform its intended audience. Recognising this focus allows the reader/learner to assess its value for their own situation. Should a researcher be undertaking a non-Cochrane, non-intervention systematic review – then perhaps Module 1 is all they would need to complete.
Navigation and online reviews:
One of the failings of the CRAAP test is that it was not designed to do an evaluation of an online platform. This left the team with the need to assess the layout and navigation of the platform. Thankfully Cochrane has kept track of the 432 people (at the time of writing) who felt compelled to provide feedback and highlight their rating of the modules at 4.6 stars (out of a possible 5).
Initial reports from the reviewing team indicated an appreciation for the level of interactivity in these modules. The modules are like an interactive book and the learner is not only gifted with a limited amount of text on the screen, but the information is interspaced with videos and quizzes to support the kinaesthetic learner. Further experience with the modules identifies some frustrations with the navigation between modules and the inflexibility in the content entry features of some quizzes.
Comparison with other available means of guidance through the systematic review process:
An ‘Environmental scan and evaluation of best practices for online systematic review resources’ was published in 2017 in the Journal of the Medical Library Association (http://jmla.pitt.edu/ojs/jmla/article/view/241) which “revealed that resources include appropriate content but are less likely to adhere to principles of online training design and interactivity”. This paper included evaluation of the earlier version of Cochrane Training “Online Learning Modules for Cochrane Authors” as well as other resources for online training in systematic review methodology.
As well as many open access and freely available online resources and guides for producing systematic reviews, there is also Cochrane Evidence Essentials (https://training.cochrane.org/essentials) - “a free online resource offering an introduction to health evidence, and how to use it to make informed health choices”.
Recognising the key role of information specialists, Module 3 of Cochrane Interactive Learning highlights the role of librarians in supporting systematic reviews: “A key part of finding eligible studies is knowing what to search for, and where and how to search for it. Information Specialists can help you with this process.”
Main results
Inspired by Cochrane's logo (https://www.cochrane.org/about-us/difference-we-make), we developed a forest plot to summarise our results. Each horizontal line represents the length of time required to complete each module and our "experienced opinion" regarding their effect. The diamond represents the combined result, our best estimate of whether the modules are effective. The diamond sits clearly to the left of the vertical line representing "no difference", therefore the evidence indicates that the intervention is beneficial. Our forest plot shows that Cochrane Interactive Learning modules are helpful for supporting those conducting a systematic review.
Authors’ conclusions
The modules are thorough in their presentation and make good use of experts' opinions and examples. The certificates awarded at the end of each module are a splendid idea, although some of the answers required may be a little pedantic. For the sake of the participants well-being, the transition from knowing, to knowing they know, is an important one and the certificates go some of the way towards that.
There is an opportunity for modules like these to evolve and become key resources for supporting the production of high-quality systematic reviews. This is especially relevant with the rapid shift to online learning and delivery of healthcare – the pandemic has been like a wrecking ball! However, there is no getting past the limited health focus and the price. We exist in times where we need to find budget savings, where resources previously thought sacrosanct are being questioned and even discarded. These decisions are becoming more difficult each year, therefore while a compelling case can be made to subscribe to the CIL modules, what resources can we relinquish to make it so? There are freely available resources that cover the same ground, such as Johns Hopkins’ Introduction to Systematic Review and Meta-Analysis, as well as a number of library guides. It is unfortunate that fiscal factors become the point of determination, but it is the world we live in. This obviously is a matter on which each individual library will make up their own minds, and some may well feel that Cochrane Interactive Learning provides sufficient value to warrant some juggling of their subscriptions budget, but it is our feeling that the matter of cost may well be the deciding factor in deliberations around this high-quality resource. If you have the opportunity to try these modules, they are well worth the investment!