Doing research in education is difficult, very difficult. Trust me. So while this review may come as a shock, I’m not that shocked.
I’m performing a RCT myself at the moment and doing it in real life situations is proving to be very hard.
From the press release:
Educational trials aimed at boosting academic achievement in schools are often uninformative, new research suggests.
The new study, published in the journal Educational Researcher, found that 40% of large-scale randomised controlled trials (RCTs) in the UK and the US failed to produce any evidence as to whether an educational intervention helped to boost academic attainment or not.
The researchers evaluated 141 trials involving more than one million students, which tested schemes ranging from whether providing free school breakfasts raises grades in Maths and English, to whether playing chess improves numeracy skills.
The trials, which were carried out by the charitable organisation the Education Endowment Foundation (EEF) in the UK and the National Center for Education Evaluation and Regional Assistance (NCEE) in the US, are expensive – with costs often exceeding £500,000.
The authors of the study argue that more research is urgently needed to understand why RCTs in education are so often uninformative.
Lead author of the research, Dr Hugues Lortie-Forgues, from the Department of Education at the University of York, UK, said: “Just like in medicine, trials of educational interventions are an important way to allow policy makers and teachers to make informed decisions about how to improve education. However, many of these trials are currently not fulfilling their main aim of demonstrating which interventions are effective and which are not.”
“Further research to investigate the reasons for this should be a priority. These organisations are trying to achieve something positive and reform is urgently needed to help them to do so.”
In recent years there have been a growing number of RCTs conducted in education. For example, in the UK, the Education Endowment Foundation (EEF) has commissioned more than 191 trials since 2012.
The researchers cite possible reasons why current trials may be ineffective, including:
- The interventions being tested may not be suitable for trial in the first place.
- Interventions may not be being correctly implemented during trials – for example due to inadequate training of teachers in the methods being tested.
- The trials themselves may be poorly designed
The authors suggest a series of changes that could make the trials more informative, including higher-standards when considering which new initiatives are trialled.
Rigorous Large-Scale Educational RCTs are Often Uninformative: Should We Be Concerned? Is published in the journal Educational Researcher. The study was carried out in collaboration with researchers at Loughborough University.
Abstract of the review:
There are a growing number of large-scale educational randomized controlled trials (RCTs). Considering their expense, it is important to reflect on the effectiveness of this approach. We assessed the magnitude and precision of effects found in those large-scale RCTs commissioned by the UK-based Education Endowment Foundation and the U.S.-based National Center for Educational Evaluation and Regional Assistance, which evaluated interventions aimed at improving academic achievement in K–12 (141 RCTs; 1,222,024 students). The mean effect size was 0.06 standard deviations. These sat within relatively large confidence intervals (mean width = 0.30 SDs), which meant that the results were often uninformative (the median Bayes factor was 0.56). We argue that our field needs, as a priority, to understand why educational RCTs often find small and uninformative effects.