There is a new Best Evidence in Brief, and this time, I picked this very relevant study from this biweekly newsletter written by Marta Pellegrini, University of Cagliari (Italy).

Educational researchers are increasingly concerned with the quality of systematic reviews because review results play a crucial role in informing policy and practice decisions. A recent study published by Thomas Nordström and colleagues assessed the risk of bias and the open science practice of education systematic reviews.

The study evaluated 88 reviews published between 2019 and 2021 and focused on the impact of K-12 interventions on student outcomes. The ROBIS (Risk Of Bias In Systematic reviews) scale was used to assess the quality and reliability of systematic reviews. The scale consists of four domains: eligibility criteria, identification and selection of the studies, data collection and study appraisal, and synthesis and findings. Practices of open science, such as protocol preregistration and availability of data, were also assessed.

Results showed that 88% of the reviews had a high risk of bias. Among  them, thirty-one reviews did not conduct a comprehensive and reproducible search. About 30 reviews did not formally evaluate the risk of bias of the original studies. A small portion of the assessed reviews (11%) were judged to have a low risk of bias. Of these, only six reviews pre-registered a protocol and three shared their data.

The authors concluded that it is crucial to implement all aspects of systematic review reporting in future research to ensure replicability. Open science has been also identified as an area needing improvement.