John Hattie has undoubtedly had a huge influence on education with Visible Learning, and I confess, on my work as well. His books and website, which bring together the results of thousands of meta-analyses, have helped make educational research accessible to teachers and policymakers. At the same time, critics have questioned his approach for years; I have found errors myself. After all, can you simply combine all those different studies? And what do those known effect sizes actually mean?
A new study by Kalmendal Bruns and colleagues, published in Educational Research Review, goes much further than previous criticisms. The researchers didn’t examine a few examples where things might go wrong. Instead, they systematically assessed the methodological quality and reproducibility of the data underlying Visible Learning. Their conclusions are quite damning. Not necessarily for the conclusions themselves, but certainly for the quality and quality control!
The researchers examined the 322 so-called influences included in the Visible Learning database. These concern influences on learning performance, ranging from feedback and homework to class size and various forms of instruction. Together, these are now based on over 2,000 meta-analyses. The researchers located and evaluated 1,375 of these meta-analyses. For 21 influences, they could not find any underlying meta-analysis, meaning they ultimately investigated 301 influences methodologically. I also noticed that some meta-analyses are indeed harder to obtain. This may be because, for example, they are part of a doctoral study that was not published as an article.
What were some of the specific issues under scrutiny? For instance, the reviewers assessed whether the studies included conform to the influence mentioned by Hattie, whether the effect size was used properly, and whether the findings could be reproduced. They also checked whether the synthesis of various kinds of research was done appropriately. And here comes the issue.
To begin with, only 18.6 per cent of the investigated influences meet all four so-called PICO criteria. This means the studied population, the intervention or exposure, the comparison, and the measured outcomes must align sufficiently with what one claims to investigate. This doesn’t mean the rest of the research is flawed, just to be clear!
That sounds logical, but in practice, things often turn out differently. For example, Hattie and his team included a meta-analysis in writing programs that actually investigated a reading intervention. They combined results on behaviour, IQ, learning performance, and other outcomes. These are not necessarily bad studies, but they do not all answer the same question. This was also what went wrong with collective teacher efficacy a few years ago.
A second problem is that different types of research are mixed together. In 27.2 per cent of the influences, researchers combined experimental and observational studies. That is less innocent than you might think. I regularly emphasise the difference between correlation and causality on this blog. For example, if students who do more homework achieve better results, that does not automatically mean more homework causes those better results. Mixing these distorts the result. Moreover, effect sizes from different research designs are not always directly comparable.
But then there is also reproducibility. The researchers attempted to reconstruct the figures from Visible Learning based on the original meta-analyses. In doing so, they looked not only at the effect size, but also at the number of studies, participants, and effects on which it was based. They stopped whenever they could not reproduce a component.
Of the 313 extraction attempts, they found the reported effect size in 179 cases. However, in only 80 of these did that effect size align with the intended research question. Ultimately, they reproduced both the effect size and the other reported numbers in only 22 extractions. At the full-influence level, they succeeded in doing so in only two of the 301.
This final statistic also warrants caution. It does not mean that the effect sizes of the other 299 influences are incorrect. The researchers did use strict standards. For example, even when the reported number of participants did not match the original source, an influence was not considered fully reproducible. In addition, the researchers could not find all the sources in which the original publication appeared. Despite these nuances, this outcome is especially disturbing for work based mostly on statistical influence.
Is this new study beyond all criticism? Of course not. Even this impressive analysis has its limitations. The researchers registered their study in advance and made their data available. Those are important strengths. However, a single researcher initially assessed the meta-analyses, with two colleagues jointly checking just over half of the assessments. The researchers also could not locate 646 meta-analyses. Only four experts performed the final assessment of the relevance of the various influences, with a moderate degree of mutual agreement. I would therefore interpret those latter results in particular with caution. However, this absolutely does not alter the fact that this study exposes a fundamental problem.
So what does this mean for Visible Learning? According to the authors, the problems are so systemic that it makes little sense to simply continue updating the existing database. Instead, they advocate for high-quality systematic reviews that answer separate, clearly defined research questions.
I think we need to keep two things separate here. On the one hand, Hattie deserves significant credit. His work has made educational research much more visible (yes, pun is intended). His work has encouraged teachers and school leaders to look beyond intuition, tradition, or the latest educational trends. He and Marzano introduced the broader educational world to evidence-based education.
On the other hand, this study makes it particularly difficult to use Visible Learning’s precise effect sizes, and especially its well-known rankings, as a reliable guideline, even though Hattie himself said in his latest book that he no longer promotes the rankings. An effect size of 0.60 is not automatically a better choice than an effect size of 0.30. That has long been a major criticism; for example, compare it with the toolkit from the EEF and Leerpunt, which is much more nuanced. But the new study shows that a lot can also go wrong when compiling those figures.
Does this mean that feedback doesn’t work? Or is this the death blow to explicit instruction, formative assessment, or other teaching approaches because they suddenly supposedly no longer have a scientific basis? Of course not. A wealth of independent research on these topics exists that doesn’t depend on Hattie’s synthesis.
It does mean that we need to be even more cautious when using a single figure from Visible Learning to defend an educational choice. People like me have been saying this for quite some time. It was practically the conclusion of our second book on myths. I once described Hattie’s work as primarily a handy phone register of educational research. The man himself told me that he had never expected his work to become so extensive. And that – had he known- he would have been stricter with quality control. I thought he had addressed this over the past few years. But my own experiences and this massive study clearly contradict that.