In 2014 Makel en Plucker found out that less than 1% of the studies published in the top journals for education research were replication studies. Did things get better afterwards? This is what Perry, Morris & Lea tried to find out. And the answer is yes, but very, very, very slightly.
| Publication Year | Total publications | Replication studies | Direct Replications | ||
|---|---|---|---|---|---|
| Freq. | Freq. | % | Freq. | % | |
| 2011 | 18,534 | 24 | 0.13 | 8 | 0.04 |
| 2012 | 19,814 | 27 | 0.14 | 8 | 0.04 |
| 2013 | 20,110 | 29 | 0.14 | 8 | 0.04 |
| 2014 | 20,009 | 39 | 0.19 | 13 | 0.06 |
| 2015 | 20,189 | 43 | 0.21 | 7 | 0.03 |
| 2016 | 21,870 | 43 | 0.20 | 13 | 0.06 |
| 2017 | 22,285 | 43 | 0.19 | 17 | 0.08 |
| 2018 | 23,542 | 52 | 0.22 | 23 | 0.10 |
| 2019 | 29,355 | 61 | 0.21 | 21 | 0.07 |
| 2020 | 30,336 | 81 | 0.27 | 27 | 0.09 |
| Total | 226,044 | 442 | 0.20 | 145 | 0.06 |
No really:
This just ain’t good. Psychology as a science has gone through a large self-correcting replication crisis, making psychology as science better.
Abstract of “A decade of replication study in education? A mapping review (2011–2020)”:
Replication studies in education are relatively rare. Of the few which are conducted, many are conceptual rather than direct replications. With so few replication studies, and many of those that are attempted producing null results, the scientific status of the evidence base for educational policy and practice is in question. Replicating Makel and Plucker’s review of the education replication literature, conducted in 2014, this paper presents a mapping review looking at rates of replication in education research from 2011 to 2020. We provide an overview of the number of replication studies by replication type, year, outcome, authorship, and journal. Our results are consistent with those of Makel and Plucker, revealing very low but gradually increasing rates of replication study in education. We discuss the role of replication in producing a robust and trustworthy evidence base for policy and practice, and some of the challenges in operationalising definitions of replication we encountered.
Reblogged this on kadir kozan.
[…] Jelte Wicherts en Janneke van de Pol, besloot dit uit te zoeken. Niet met een “nieuw model”, maar met iets wat in de onderwijswetenschap nog te weinig gebeurt: een directe replicatie. Ze reconstrueerden het klassieke experiment van David Wood en collega’s uit 1978, dat vaak wordt […]
[…] Jelte Wicherts and Janneke van de Pol, decided to find out. Not by proposing a “new model”, but by doing something that still happens far too rarely in education research: a direct replication. They reconstructed the classic 1978 experiment by David Wood and colleagues, often cited as the […]
[…] en sterk afhankelijk zijn van context. Wil dit dan zeggen dat er geen probleem is? Zeker niet. Niet omdat onderwijsonderzoek uitzondelijk vaak faalt, wel omdat het gewoon relatief weinig gebeurt volgens ander […]
[…] In a parallel study published in Nature, Andrew Tyner and colleagues examined replicability. They tested whether the results could be replicated in new studies. Here, we see a different picture. In this large SCORE project, replication success hovers around 50%, and this also applies to education research. Not exceptionally good, but not exceptionally bad either. It is what you would expect in social science: effects that are often smaller and strongly dependent on context. Does this mean there is no problem? Certainly not. Not because education research fails unusually often, but because replication still happens relative… […]