Yesterday, Paul Kirschner responded to my blog post discussing research I was critical of, ending with the words: “Again, lots of time and money invested to learn virtually nothing.” Paul has a point. And this new umbrella review will not make him, or you as a reader, feel any more hopeful.

Imagine fifteen meta-analyses, together based on hundreds of studies of project-based learning. All eight meta-analyses looking at academic achievement find a positive effect. The reported effects range from a modest d = 0.277 to a spectacular d = 1.28. You would think that by now we have a pretty good idea of whether project-based learning works. But it turns out that these meta-analyses tell us next to nothing.

In this new umbrella review, Farshad and Fortin put those fifteen meta-analyses side by side and also assessed their methodological quality. The result is pretty painful and sobering: all fifteen were rated ‘Critically Low’. Not a single meta-analysis even made it to ‘Low’, let alone ‘Moderate’ or ‘High’.

Does this mean that project-based learning does not work? No, we cannot conclude that from this review either. All eight relevant meta-analyses point in the same positive direction. So there does seem to be something there. But we know much less about the actual size of the effect and the conditions under which it occurs than fifteen meta-analyses might lead you to expect.

So why do all these meta-analyses receive such a damning verdict from Farshad and Fortin? None of the fifteen had preregistered a protocol or provided a justified list of excluded studies. Thirteen out of fifteen failed to adequately assess the risk of bias in the original studies, and fourteen did not sufficiently take that risk of bias into account when interpreting their results.

And if you are thinking that perhaps all those meta-analyses were based on the same studies, think again. They barely recycle the same research. For thirteen of the meta-analyses, Farshad and Fortin were able to identify the underlying studies (which also says something about those other two meta-analyses…). They found 351 unique studies. Only twelve appeared in more than one meta-analysis, and not a single study appeared in more than two.

So perhaps we at least know when project-based learning works better? Hardly. Results regularly contradict each other when it comes to educational level, subject area and duration. Smaller groups and technological support seem to produce somewhat more consistently positive results, but of course those conclusions also come from meta-analyses of very low methodological quality.

The real conclusion of this review is therefore rather sobering. We have hundreds of studies, fifteen meta-analyses, and results that repeatedly point in the same positive direction when it comes to academic achievement. Great. But after all that research, you would expect us to have a much better idea by now of how well it works and under what conditions.

It may not be virtually nothing, but it is not that much more either.