Category Archives: Review

There is a new best evidence in brief and in the new edition there is a bit of a meta-subject as it’s not about research but about research about research (you’ll have to read that twice, I guess).

The use of randomized controlled trials (RCTs) in education research has increased over the last 15 years. However, the use of RCTs has also been subject to criticism, with four key criticisms being that it is not possible to carry out RCTs in education; the research design of RCTs ignores context and experience; RCTs tend to generate simplistic universal laws of “cause and effect”; and that they are descriptive and contribute little to theory.

To assess these four key criticisms, Paul Connolly and colleagues conducted a systematic review of RCTs in education research between 1980 and 2016 in order to consider the evidence in relation to the use of RCTs in education practice.

The systematic review found a total of 1,017 RCTs completed and reported between 1980 and 2016, of which just over three-quarters have been produced in the last 10 years. Just over half of all RCTs were conducted in North America and just under a third in Europe. This finding addresses the first criticism, and demonstrates that, overall, it is possible to conduct RCTs in education research.

While the researchers also find evidence to oppose the other key criticisms, the review suggests that some progress remains to be made. The article concludes by outlining some key challenges for researchers undertaking RCTs in education.

Background: The use of randomised controlled trials (RCTs) in education has increased significantly over the last 15 years. However, their use has also been subject to sustained and rather trenchant criticism from significant sections of the education research community. Key criticisms have included the claims that: it is not possible to undertake RCTs in education; RCTs are blunt research designs that ignore context and experience; RCTs tend to generate simplistic universal laws of ‘cause and effect’; and that they are inherently descriptive and contribute little to theory.

Purpose: This article seeks to assess the above four criticisms of RCTs by considering the actual evidence in relation to the use of RCTs in education in practice.

Design and methods: The article is based upon a systematic review that has sought to identify and describe all RCTs conducted in educational settings and including a focus on educational outcomes between 1980 and 2016. The search is limited to articles and reports published in English.

Results: The systematic review found a total of 1017 unique RCTs that have been completed and reported between 1980 and 2016. Just over three quarters of these have been produced over the last 10 years, reflecting the significant increase in the use of RCTs in recent years. Overall, just over half of all RCTs identified were conducted in North America and a little under a third in Europe. The RCTs cover a wide range of educational settings and focus on an equally wide range of educational interventions and outcomes. The findings not only disprove the claim that it is not possible to do RCTs in education but also provide some supporting evidence to challenge the other three key criticisms outlined earlier.

Conclusions: While providing evidence to counter the four criticisms outlined earlier, the article suggests that there remains significant progress to be made. The article concludes by outlining some key challenges for researchers undertaking RCTs in education.

There was a new best evidence in brief past week and this item isn’t that great news:

An evaluation conducted for the Education Endowment Foundation in the UK looked at whether the  Good Behaviour Game (GBG) improved students’ reading skills and behavior.

The GBG intervention is a classroom management approach designed to improve student behavior and build confidence and resilience. The game is played in groups and rewards students for good behavior. More than 3,000 Year 3 (equivalent to second grade in the U.S.) students from 77 UK schools took part in a randomized controlled trial of GBG over two years. Around a quarter of the students in the schools were eligible for free school meals, around a fifth were students with special educational needs, and 23% had English as an additional language.

The analysis indicated that, on average, GBG had no significant impact on students’ reading skills (effect size = +0.03) or their behavior (concentration, disruptive behavior, and pro-social behavior) when compared to the control group students. However, there was some tentative evidence that boys at risk of developing conduct problems showed improvements in behavior.

While I was on leave in the States I kept reading interesting studies and news articles and I was really impressed by both this study and this article about the study by Ed Yong for The Atlantic. Both the authors of the study and Ed know how touchy the subject of genes and intelligence can be. That is why the researchers wrote an accompanying FAQ that explains what they found and what it means.

What I like in The Atlantic article is the good nuanced reporting, such as:

This isn’t to say that staying in school is “in the genes.” Each genetic variant has a tiny effect on its own, and even together, they don’t control people’s fates. The team showed this by creating a “polygenic score”—a tool that accounts for variants across a person’s entire genome to predict how much formal education they’re likely to receive. It does a lousy job of predicting the outcome for any specific individual, but it can explain 11 percent of the population-wide variation in years of schooling.

And the explaining what it means

“…Now, consider that household income explains just 7 percent of the variation in educational attainment, which is less than what genes can now account for. “Most social scientists wouldn’t do a study without accounting for socioeconomic status, even if that’s not what they’re interested in,” says Harden. The same ought to be true of our genes.”

Do read the complete article in The Atlantic or even better, the original study:

Here we conducted a large-scale genetic association analysis of educational attainment in a sample of approximately 1.1 million individuals and identify 1,271 independent genome-wide-significant SNPs. For the SNPs taken together, we found evidence of heterogeneous effects across environments. The SNPs implicate genes involved in brain-development processes and neuron-to-neuron communication. In a separate analysis of the X chromosome, we identify 10 independent genome-wide-significant SNPs and estimate a SNP heritability of around 0.3% in both men and women, consistent with partial dosage compensation. A joint (multi-phenotype) analysis of educational attainment and three related cognitive phenotypes generates polygenic scores that explain 11–13% of the variance in educational attainment and 7–10% of the variance in cognitive performance. This prediction accuracy substantially increases the utility of polygenic scores as tools in research.

I’m much in favor of open access and I’m so glad this new review by Anne Castles, Kathleen Rastle and Kate Nation is free to read for everybody out there. Why is this great news? Well, this review is a very nuanced overview of anything related to reading acquisition. In a field that has known several reading wars, this is no little thing to try to achieve. But will it end the reading wars? Looking at the history described at the beginning of the article I’m not so sure, but if so those wars will continue despite this article.

Do note, because of the big scope of this article, it’s a really long read. But well worth of your attention.

Abstract of this review:

There is intense public interest in questions surrounding how children learn to read and how they can best be taught. Research in psychological science has provided answers to many of these questions but, somewhat surprisingly, this research has been slow to make inroads into educational policy and practice. Instead, the field has been plagued by decades of “reading wars.” Even now, there remains a wide gap between the state of research knowledge about learning to read and the state of public understanding. The aim of this article is to fill this gap. We present a comprehensive tutorial review of the science of learning to read, spanning from children’s earliest alphabetic skills through to the fluent word recognition and skilled text comprehension characteristic of expert readers. We explain why phonics instruction is so central to learning in a writing system such as English. But we also move beyond phonics, reviewing research on what else children need to learn to become expert readers and considering how this might be translated into effective classroom practice. We call for an end to the reading wars and recommend an agenda for instruction and research in reading acquisition that is balanced, developmentally informed, and based on a deep understanding of how language and writing systems work.

The past few weeks there were so many interesting reports and studies published that it’s sheer impossible to keep track while teaching. In this post I want to highlight what you really shouldn’t miss.

First there is the report that  Daniel Muijs and Christian Bockhove wrote on metacognition and self-regulated learning for the EEF:

This guidance draws on a review of the evidence about self-regulated learning and metacognition led by Professor Daniel Muijs and Dr Christian Bokhove (University of Southampton). It is not a new study in itself, but rather is intended as an accessible overview of existing research with clear, actionable guidance. More information about the review and the process is at the end of the review. Some key references are included here; for those wishing to explore the subject in more depth, the forthcoming report will contain a more comprehensive reference section.

This is a summary of the more important recommendations:

Next up is this Oxford report on why closing the word gap is crucial. It’s a pretty big report, so if you only have limited time, do read the first part with the overview of the evidence, although the whole report is important. You want to know why, well read this excerpt:

Almost all primary school teachers surveyed by OUP felt that the word gap resulted in weaker comprehension skills and slower than expected progress in reading and writing. Secondary school teachers overwhelmingly stated that a low vocabulary affects a child’s progress through secondary school across a wide range of subjects. The vast majority of those surveyed (86% of primary teachers and 80% of secondary teachers) responded that they thought it was very or extremely challenging for pupils with a limited vocabulary to read national test papers. Consequently, 82% primary teachers and 79% of secondary teachers noted that these pupils are likely to get worse results in national tests.

And also:

Self esteem, behaviour and a child’s ability to make friends were all felt to be negatively affected by low levels of vocabulary.

A third report isn’t actually a report but Daniel Willingham who is making a plea about what teacher trainees should learn about educational psychology.  You can check:

• Article in Education Next
• More technical (and more complete) article in Mind, Brain, Education
• Podcast Interview
• Audio from talk at ResearchEd

This abstract from the article in Mind, Brain, Education summarizes the main idea:

Although most teacher education programs include instruction in the basic science of psychology, practicing teachers report that this preparation has low utility. Researchers have considered what sort of information from psychology about children’s thinking, emotion, and motivation would be useful for teachers’ practice. Here, I take a different tack. I begin by considering three varieties of statements in basic science: empirical observations, theoretical statements, and epistemic assumptions. I suggest that the first of these can support classroom application, but the latter two cannot. I use that conclusion as a starting point for considering the instruction of prospective teachers in psychology.

I for myself am not that sure if I agree with Daniel on this line of thinking, but I do think it’s an important debate to have and therefor well worth the read.