*update*: check John Hattie’s reaction under this post. It was indeed a mistake, and it will be corrected.
*Not updated*: I’m writing this on November 4th, 2025, and it’s still not corrected!
I’m a nerd on effect sizes and have been following the Visible Learning MetaX website for a long time. I was surprised to see a considerable drop in the effect size of Collective Teacher Efficacy. While in October, it still was 1.34-136, now it suddenly dropped to 1.01-1.12:
Many average effect sizes in Hattie’s work have been dropping because science is handling publication bias better, so this could be the case for collective teacher efficacy, too. But as you can see, the drop is caused by one new meta-analysis with an extremely low effect size.
Ok, let’s check this meta-analysis. Did the researchers make a mistake, or did they find new studies on the topic? No, I’ve read the research, and Çogaltay & Boz did an excellent job. But… the meta-analysis is not on the learning effect of collective teacher efficacy! Instead, they examined school leaders’ impact on collective teacher efficacy, which is entirely different.
If you look at who the 25645 participants in the 47 studies were, you can see that they weren’t students but teachers!
So, I think somebody in the Visible Learning MetaX team has made a mistake and yes, I did mail them.
Abstract of the study:
The existing literature provides compelling evidence on the effect of school leadership on the collective efficacy of teachers, which is highly correlated with teaching–learning outcomes. However, the results of studies that examined the relationship between these variables have not reached a consensus. Therefore, the main objective of this study is to determine the extent to which school leadership influences collective teacher efficacy through a meta-analysis. The study identified 47 research articles with a total of 25,645 respondents. The results suggest a large-level effect of school leadership behaviors on collective teacher efficacy. Moreover, three variables; leadership style, country culture, and school level, moderate the effect of school leadership on collective teacher efficacy. Especially, the impact of instructional leadership is stronger than those of transformational and other types of leadership. In addition, the magnitude impact of school leadership on collectivist culture is larger than that of individualistic culture. Furthermore, the findings suggest that the main effect size of school leadership on teacher collective efficacy in high schools is greater than those in other school levels. Lastly, the paper presents the implications and limitations of this study as well as recommendations for future research.
[…] Wil dat zeggen dat er niets fout kan gaan met onderzoek? Helemaal niet. Wil het zeggen dat ook grote namen of instituten geen fouten kunnen maken? Nee, zeker niet. Zo vond ik dit weekend een behoorlijke fout bij de Visible Learning MetaX website waar een nieuwe me…. […]
You are so right Pedro. I will take it out in the next update and the effect goes to 1.34.
So appreciate when you and others email with me with errors and corrections. Maintaining the data base to minimize the errors is tough. Sometimes it is conversion of r to d, sometime some argue with my categorization, and (like you) some pick up metas that should not be there. This is a working document and every one of these corrections/updates/queries is more than welcome (and there is a link in MetaX to raise issues). Thanks Pedro
John Hattie
[…] *update: John Hattie reageerde op mijn Engelstalige blog en het is inderdaad een fout. De effect size wordt t… […]
[…] Does that mean that nothing can go wrong with research? Not at all. Does it mean that big names or institutes cannot make mistakes? No, certainly not. This weekend, I found a grave error on the Visible Learning MetaX website, where a new meta-analysis… […]
[…] When someone claims that correcting misinformation doesn’t work “on average,” it’s good to ask: what exactly are you talking about? Because an average that combines opposite effects sometimes doesn’t tell you anything at all. ( or sometimes people even combine things that have nothing to do with each other ). […]