If you work in education, you will recognize the dilemma: you read about a great new approach that works exceptionally well somewhere, but you immediately wonder: will that work at my school? And especially, will it work for my students, my teachers, and my context?

The What Works Clearinghouse (WWC) in the United States does just that: investigate what works in education with very strict criteria. But for a long time their focus was mainly on internal validity – simply put, whether the effect found is really due to the intervention itself. Less attention was paid to external validity or whether those results also apply in other situations and for other students.

In her research, Betsy Wolf rightly asks the question: Do we actually know enough about for whom certain interventions work and under what circumstances? She, therefore, delved into the WWC data and created clear overview maps of which students, schools, and outcomes are now included in all that research.

The results were striking. The WWC has collected a lot of data on students and schools, but important information about students’ backgrounds is often missing. Also, some groups, such as students with specific disadvantages or disabilities, have been studied only to a limited extent. Furthermore, there is an overrepresentation of research from large, urban schools and certain states, so one must be cautious about generalizing to smaller or rural schools.

Another interesting conclusion from the research is that certain types of outcomes – such as arithmetic and reading skills – have been extensively researched, while social-emotional development or school climate, for example, remain significantly underexposed.

In short: “What works?” is a useful question, but it only becomes truly relevant when we also ask, “For whom does it work, and in what context?” The better we understand those nuances and the more fully research takes those nuances into account, the more useful it becomes for everyone in the education field.

It is a concern that I endorse and that I also recognize among Leerpunt’s (international) partners. You will notice that such information is already present in the EEF Toolkits that we have retranslated. This year, the scientific basis of the toolkit will be further expanded with almost 2000 studies precisely to meet this need.

At the same time I also notice that context can also be used to completely dismiss educational research. A kind of extreme form of evidence-informed side to ensure that there is no evidence left. That certainly cannot be the intention either…

Abstract of the study :

The What Works Clearinghouse (WWC) at the Institute of Education Sciences reviews rigorous research on educational practices, policies, programs, and products with a goal of identifying “what works” and making that information accessible to the public. One critique of the WWC is the need to more closely examine “what works” for whom, in which settings, and for which outcomes. The purpose of this article is to create evidence maps of the WWC study data to display the student populations, settings, and outcomes included in the evidence base. Findings show that the WWC study data include a broad range of students, settings, and outcomes relevant to education in the US, yet missing data on sample characteristics are a concern. The WWC evidence base also includes relatively few studies focused on specific student populations or some outcome domains. Finally, findings suggest that “what worked,” or the extent to which studies found favorable results, meaningfully depended on which outcomes were examined in the studies. The findings have implications for researchers, funders of research, and the WWC.