These are difficult times for people who do science, blog about science, and defend science. I do all three. Daniel Muijs wrote this blog post over the weekend defending educational research. He starts by acknowledging that there are indeed problems (certainly), but that there is also a lot of good research being done (also certainly). He even provides a list of recent, excellent studies. Yet at the same time, among scientists in London last Saturday, it was mainly the problems that were being discussed.

I am now wondering whether I should give an overview of those problems. It is something I have been struggling with for a while. Sometimes I write about research, such as this umbrella review of 15 meta-analyses, and I worry that I am feeding anti-science sentiment. At the same time, I think it would be wrong not to discuss problems when they occur and when sound research can demonstrate them. The irony is that, in this case, it is sound research that exposes problems in other research. You could even argue that this shows science working as it should, even as the challenges keep getting bigger and bigger.

• Too much research, too little good research. The pressure to publish still rewards numbers of papers, citations, and publications, preferably in the most highly cited journals. All this encourages research that offers new insights and, above all, is publishable. This does not necessarily mean important, robust, or cumulative knowledge, even though that is precisely what we need.
• A lot of methodologically weak research. It is a personal peeve having lately read so many papers plagued by these issues. Consider small sample sizes, low statistical power, poorly validated measures, weak control groups, limited intervention periods, and more. In education research, of course, there is also the contextual sensitivity, making this often complicated.
• Publication bias and questionable research practices. HARKing, p-hacking, outcome switching and selective reporting can systematically make the scientific literature look rosier than reality. Stuart Ritchie illustrated this very well in his book Science Fictions.
• Replication remains a problem. It is one of my hobby horses, I know, but there really is too little replication in educational research. This is mainly because replications bring little prestige. So we produce lots of new claims while checking relatively few old ones.
• Syntheses are only as good as what goes into them. This ties in closely with my recent post on PBL. Ten or fifteen meta-analyses do not automatically strengthen a conclusion when the original primary studies are weak, the interventions differ substantially, or the same studies keep appearing in those meta-analyses.
• Peer review is buckling under the volume. More papers mean you need ever more reviewers and editors, while, to be clear, reviewing remains largely unpaid and often offers little satisfaction. All of this makes thorough quality control increasingly difficult. Over the past few months, I have found myself thinking several times: why wasn’t this a desk reject?
• Paper mills and outright fraud. By now, we have arrived at problems affecting science as a whole, although educational research is certainly not immune. Nature described paper mills last year as a now flourishing industry and cited an estimate of at least 400,000 papers published between 2000 and 2022 showing signs of paper-mill production.

Many of these problems are older than you might think, as 15 years of myth-busting have taught me. But AI is making all of them much more visible and worse. Generative AI lowers the effort and cost involved in producing mediocre or worthless papers. Open datasets can be automatically trawled for associations, after which an LLM can rapidly turn the results into an article that looks, above all, deceptively respectable. Recent research estimates that in 2025 alone there was an excess of around 12,000 such publications based on open health data. And then it can start to look attractive to use AI for peer review too, something some journals are already experimenting with.

You might read all this and think: haven’t we known about these problems for quite some time? Yes, we have. People have been writing about publication bias, replication problems, p-hacking and the perverse effects of publication pressure for years. So quite a few of the things I have listed above are anything but new. What may be new is the scale at which some of these problems can now start to occur.

We already had a scientific system that, in some places, rewards quantity and novelty too strongly. As a result, that system also produced mediocre and, at times, downright poor research. Generative AI has now drastically reduced the effort and cost required to produce much more of it. At the same time, the human capacity to read, review, replicate and ultimately assess all that research has not increased at the same speed. I now read more research for this blog, only to find it increasingly difficult to choose a study worth writing about.

And yet I do not want to end this post with the message that science can no longer be trusted, because I agree with what Daniel wrote in his blog. Quite the opposite. The umbrella review I wrote about earlier may itself be one of the best examples. Scientists critically examined what other scientists had done and concluded that we know far less with confidence than those fifteen meta-analyses might initially have suggested. That is not a failure of science. That is science.

And, to be absolutely clear, over the past few months I have also seen fantastic work by hard-working scientists who, sometimes against their better judgement, refuse to take the easy route. Scientists who also use AI because they are not living in another world. I use it too. But they remain in the driver’s seat.

I am also a strong supporter of the open-science movement and of the importance of preregistration, even though I sometimes get pushback on this from other scientists. These are precisely the kinds of encouraging developments that show we are working on many of these problems.

It is just that the work of separating the wheat from the chaff is becoming ever greater. If you want to defend science, you have to be willing to acknowledge its problems. To me, that is part of being a scientist. But anyone who points out these problems also knows that today any criticism can be appropriated and turned into an argument against science itself. Unedifying stories such as the Jason Arday case certainly do not help either. Sometimes it seems as if science is particularly keen on shooting itself in the foot.

That is something I struggle with. All suggestions are welcome.