Educational meta-analyses are often too optimistic

No, this won’t be another devastating blog post like the one about the recent study on Hattie. But the research I found via Christian Bokhove is extremely important for putting educational meta-analyses into perspective.

Meta-analyses are often seen as one of the strongest forms of scientific evidence. It’s a pretty simple idea: if you combine the results of several studies, you likely get a more reliable picture than if you look at a single study. I often say: ‘one study does not make an insight’. That is true, provided the underlying studies are good enough, and you combine them in a sound way. For we have seen on several occasions that things can go quite wrong with meta-analyses as well.

In a new study published in Review of Educational Research, Luke Rowe investigated how reliable meta-analyses in educational research actually are. And the results are not exactly reassuring. But, and I find this important, they do not mean that we should simply discard all those meta-analyses.

A meta-analysis of meta-analyses, but with a difference

Rowe reviewed 76 articles containing meta-analyses published between 2012 and 2022 in Review of Educational Research. This is one of the most prestigious journals in educational research. So, we can expect the published meta-analyses to be of high quality.

Together, these articles contained 362 separate meta-analytic effects. For 189 of these, drawn from 53 articles, the researcher had sufficient data to perform new calculations.

In doing so, he compared the originally reported effect sizes with new calculations using classical statistical methods on the one hand and a relatively new approach, Robust Bayesian Model-Averaged Meta-Analysis (RoBMA), on the other.

This approach combines several statistical models and, among other things, tries to account for publication bias. That last point is crucial. Research that produces statistically significant results tends to get published more frequently than research that finds no effect or statistically insignificant results. Therefore, if you use only published research, you may get an overly optimistic estimate of the effects.

The effects are often smaller than originally thought

This seems to be the case here as well. In 94% of the reanalysed experimental effects, the initially reported effect size was larger than the newly calculated effect size using RoBMA. In observational research, which examines correlations between various characteristics, this was true for about 79% of the correlations. And in some cases, the differences were substantial. Depending on the comparison, the originally reported mean effects were between 29% and 267% larger than the recalculated estimates.

Nevertheless, one thing should be pointed out right away. This does not mean that 94% of meta-analyses are incorrect. Nor does it mean that the educational interventions analysed do not work at all. The main thing the results show is that effect sizes are often overestimated when compared with estimates from a statistical method that attempts to account more thoroughly for possible biases.

An educational intervention that shows a positive effect in a meta-analysis may still have a positive effect. Nevertheless, that effect may be smaller than it initially seemed. This can make quite a difference when comparing educational interventions using effect sizes.

This study also doesn’t demonstrate that the direction of the effects is always preserved. The researcher focuses primarily on the magnitude of the estimates. Therefore, we cannot simply conclude that all original conclusions remain valid.

Another problem: many studies are too small

In addition, Rowe assessed the statistical power of the experiments included in the meta-analyses. Power is defined as the likelihood of finding an effect if an effect exists. Based on the calculated effect size, only 4.21% of the experimental meta-analyses met the standard level of 80% power. In a sample of over 10,000 experimental effects, only 29% met the standard.

That sounds rather dramatic, but here too, some nuance is needed. These power estimates are retrospective and based on estimated effect sizes. If the latter decrease, the estimated power will also decrease. These two findings are therefore partly related.

This highlights a familiar problem in educational research: many individual studies are relatively small, while the effects being investigated are often small as well. As a result, findings can vary considerably, and random outliers can play a major role. Another striking finding is that only 55.3% of the meta-analyses provided enough data for reanalysis. After contacting the authors, this figure increased to around 70%. Only 5% made the analysis code needed for reanalysis available.

Not all educational contexts are the same

Moreover, Rowe highlights the enormous heterogeneity between studies. In the experimental meta-analyses, the median I² was above 80%, while in the observational meta-analyses, it was even above 90%. To explain briefly: Higgins’ I² estimates the percentage of variation between studies that is due to heterogeneity rather than chance. It is therefore a statistical measure that indicates what proportion of the differences between effect estimates in the included studies can be attributed to actual differences between studies, rather than to sampling errors.

We cannot conclude from this that an approach works differently in 80% of schools. It primarily means that there is a great deal of variation between the results of the different studies. But education is simply complex. Something that works for young children does not necessarily work in the same way for adolescents. An intervention can turn out differently depending on, say, students’ prior knowledge, how it is implemented, the context in which it takes place, etc.

It does, however, help us understand why we need to be careful with a single average effect size. An average may be helpful, but it does not necessarily tell us who an intervention works for, when, and under what circumstances.

Wait, is that new calculation automatically better then?

So far, it sounds as if the original meta-analyses were too optimistic and the new calculations finally show us the correct effects. But that would, in turn, be too simplistic a conclusion. RoBMA is an interesting and promising method, but not infallible. The method attempts to correct for publication bias, among other things, but can also overcorrect in some cases.

Consider, for example, instances where small studies examine a treatment that is more intensive than the treatment in large-scale studies. In such scenarios, small studies might be discovering bigger effects. A statistical model can interpret this as publication bias, but there could also be another reason. I wasn’t aware of this myself, as I’m not very familiar with the technique. Luckily, Rowe points it out himself.

Moreover, the paper examines meta-analyses from only one journal, published between 2012 and 2022. The findings are important, but we cannot extrapolate them to all meta-analyses ever published in education.

One more thing to note. The original effect sizes are compared with alternative statistical estimates. This does not suddenly give us the true effect sizes of all the interventions under consideration. For that, we would also need, for example, large-scale, independent replication studies.

What are we supposed to do with this now?

Let’s get back to the beginning of this post. This meta-meta-analysis adds important nuances. At the same time, I certainly do not think it makes meta-analyses useless. They’re still a particularly handy tool for understanding what we know from research.

A meta-analysis can help us identify the general direction of an effect. But this study confirms that its exact magnitude may be less certain than we would like. Another confirmation worth repeating: we have to keep evaluating the quality of the underlying studies, the differences between them, and how their findings have been combined.

For evidence-informed education, this is not a reason to pay less attention to research. It is a reason to look at research even more carefully.

Leave a Reply