Category Archives: Review

Are the criticisms about randomized controlled trials in education correct? (Best Evidence in Brief)

There is a new best evidence in brief and in the new edition there is a bit of a meta-subject as it’s not about research but about research about research (you’ll have to read that twice, I guess).

The use of randomized controlled trials (RCTs) in education research has increased over the last 15 years. However, the use of RCTs has also been subject to criticism, with four key criticisms being that it is not possible to carry out RCTs in education; the research design of RCTs ignores context and experience; RCTs tend to generate simplistic universal laws of “cause and effect”; and that they are descriptive and contribute little to theory.
To assess these four key criticisms, Paul Connolly and colleagues conducted a systematic review of RCTs in education research between 1980 and 2016 in order to consider the evidence in relation to the use of RCTs in education practice.
The systematic review found a total of 1,017 RCTs completed and reported between 1980 and 2016, of which just over three-quarters have been produced in the last 10 years. Just over half of all RCTs were conducted in North America and just under a third in Europe. This finding addresses the first criticism, and demonstrates that, overall, it is possible to conduct RCTs in education research.
While the researchers also find evidence to oppose the other key criticisms, the review suggests that some progress remains to be made. The article concludes by outlining some key challenges for researchers undertaking RCTs in education.
I want to add this abstract too:

Background: The use of randomised controlled trials (RCTs) in education has increased significantly over the last 15 years. However, their use has also been subject to sustained and rather trenchant criticism from significant sections of the education research community. Key criticisms have included the claims that: it is not possible to undertake RCTs in education; RCTs are blunt research designs that ignore context and experience; RCTs tend to generate simplistic universal laws of ‘cause and effect’; and that they are inherently descriptive and contribute little to theory.

Purpose: This article seeks to assess the above four criticisms of RCTs by considering the actual evidence in relation to the use of RCTs in education in practice.

Design and methods: The article is based upon a systematic review that has sought to identify and describe all RCTs conducted in educational settings and including a focus on educational outcomes between 1980 and 2016. The search is limited to articles and reports published in English.

Results: The systematic review found a total of 1017 unique RCTs that have been completed and reported between 1980 and 2016. Just over three quarters of these have been produced over the last 10 years, reflecting the significant increase in the use of RCTs in recent years. Overall, just over half of all RCTs identified were conducted in North America and a little under a third in Europe. The RCTs cover a wide range of educational settings and focus on an equally wide range of educational interventions and outcomes. The findings not only disprove the claim that it is not possible to do RCTs in education but also provide some supporting evidence to challenge the other three key criticisms outlined earlier.

Conclusions: While providing evidence to counter the four criticisms outlined earlier, the article suggests that there remains significant progress to be made. The article concludes by outlining some key challenges for researchers undertaking RCTs in education.

1 Comment

Filed under Education, Research, Review

What works and doesn’t work with instructional video, a new short overview

There is a special issue of Computers in Human Behavior on learning from video and in their Editorial Fiorella and Mayer give an overview of effective and ineffective methods that are being trialed in the special issue:

What are the effective methods?

…two techniques that appear to improve learning outcomes with instructional video are segmenting—breaking the video into parts and allowing students to control the pace of the presentation—and mixed perspective—filming from both a first-person perspective and third-person.

And what isn’t worth the effort?

…some features that do not appear to be associated with improved learning outcomes with instructional video are matching the gender of the instructor to the gender of the learner, having the instructor’s face on the screen, inserting pauses throughout the video, and adding practice without feedback.

Abstract of the editorial:

In this commentary, we examine the papers in a special issue on “Developments and Trends in Learning with Instructional Video”. In particular, we focus on basic findings concerning which instructional features improve learning with instructional video (i.e., breaking the lesson into segments paced by the learner; recording from both first- and third-person perspectives) and which features or learner attributes do not (i.e., matching the instructor’s gender to the learner’s gender; having the instructor’s face on the screen; adding practice without feedback; inserting pauses throughout the video; and spatial ability). In addition, we offer recommendations for future work on designing effective video lessons.

Leave a comment

Filed under Education, Research, Review, Technology

Not so good news about the Good Behavior Game (Best Evidence in Brief)

There was a new best evidence in brief past week and this item isn’t that great news:

An evaluation conducted for the Education Endowment Foundation in the UK looked at whether the  Good Behaviour Game (GBG) improved students’ reading skills and behavior.
The GBG intervention is a classroom management approach designed to improve student behavior and build confidence and resilience. The game is played in groups and rewards students for good behavior. More than 3,000 Year 3 (equivalent to second grade in the U.S.) students from 77 UK schools took part in a randomized controlled trial of GBG over two years. Around a quarter of the students in the schools were eligible for free school meals, around a fifth were students with special educational needs, and 23% had English as an additional language.
The analysis indicated that, on average, GBG had no significant impact on students’ reading skills (effect size = +0.03) or their behavior (concentration, disruptive behavior, and pro-social behavior) when compared to the control group students. However, there was some tentative evidence that boys at risk of developing conduct problems showed improvements in behavior.

Leave a comment

Filed under Education, Research, Review

Good read: An Enormous Study of the Genes Related to Staying in School

While I was on leave in the States I kept reading interesting studies and news articles and I was really impressed by both this study and this article about the study by Ed Yong for The Atlantic. Both the authors of the study and Ed know how touchy the subject of genes and intelligence can be. That is why the researchers wrote an accompanying FAQ that explains what they found and what it means.

What I like in The Atlantic article is the good nuanced reporting, such as:

This isn’t to say that staying in school is “in the genes.” Each genetic variant has a tiny effect on its own, and even together, they don’t control people’s fates. The team showed this by creating a “polygenic score”—a tool that accounts for variants across a person’s entire genome to predict how much formal education they’re likely to receive. It does a lousy job of predicting the outcome for any specific individual, but it can explain 11 percent of the population-wide variation in years of schooling.

And the explaining what it means

“…Now, consider that household income explains just 7 percent of the variation in educational attainment, which is less than what genes can now account for. “Most social scientists wouldn’t do a study without accounting for socioeconomic status, even if that’s not what they’re interested in,” says Harden. The same ought to be true of our genes.”

Do read the complete article in The Atlantic or even better, the original study:

Abstract of the study:
Here we conducted a large-scale genetic association analysis of educational attainment in a sample of approximately 1.1 million individuals and identify 1,271 independent genome-wide-significant SNPs. For the SNPs taken together, we found evidence of heterogeneous effects across environments. The SNPs implicate genes involved in brain-development processes and neuron-to-neuron communication. In a separate analysis of the X chromosome, we identify 10 independent genome-wide-significant SNPs and estimate a SNP heritability of around 0.3% in both men and women, consistent with partial dosage compensation. A joint (multi-phenotype) analysis of educational attainment and three related cognitive phenotypes generates polygenic scores that explain 11–13% of the variance in educational attainment and 7–10% of the variance in cognitive performance. This prediction accuracy substantially increases the utility of polygenic scores as tools in research.

Leave a comment

Filed under Education, Research, Review

Again: the power of forgetting

People who read my book or who saw a presentation probably know it already, but I’m a big fan of Ebbinghaus who described the forgetting curve in 1885. His influence on things such as spaced repetition – one of the most effective ways to remember stuff – is big. Spaced repetition already shows the power of forgetting, this announcement of a talk by Bjork, Robert A. that is, gives a good short overview:

Contextual clues play a role in what people are able to store and retrieve from their memory, says Robert A. Bjork, PhD, distinguished research professor in the department of psychology at the University of California, Los Angeles. A change in context can cause forgetting, but it can also change–and enrich–how information is encoded and retrieved, which can enhance learning. Bjork defines forgetting as “a decrease in how readily accessible some information or procedure is at a given point in time.” For example, some items may be strongly imprinted in our memories (referred to as “strong storage strength”)–such as a childhood phone number–but may be difficult to retrieve quickly due to the length of time since that information has been accessed (“weak retrieval strength”).


Filed under Education, Psychology, Research, Review

So glad this review is open access: “Ending the Reading Wars: Reading Acquisition From Novice to Expert”

I’m much in favor of open access and I’m so glad this new review by Anne Castles, Kathleen Rastle and Kate Nation is free to read for everybody out there. Why is this great news? Well, this review is a very nuanced overview of anything related to reading acquisition. In a field that has known several reading wars, this is no little thing to try to achieve. But will it end the reading wars? Looking at the history described at the beginning of the article I’m not so sure, but if so those wars will continue despite this article.

Do note, because of the big scope of this article, it’s a really long read. But well worth of your attention.

Abstract of this review:

There is intense public interest in questions surrounding how children learn to read and how they can best be taught. Research in psychological science has provided answers to many of these questions but, somewhat surprisingly, this research has been slow to make inroads into educational policy and practice. Instead, the field has been plagued by decades of “reading wars.” Even now, there remains a wide gap between the state of research knowledge about learning to read and the state of public understanding. The aim of this article is to fill this gap. We present a comprehensive tutorial review of the science of learning to read, spanning from children’s earliest alphabetic skills through to the fluent word recognition and skilled text comprehension characteristic of expert readers. We explain why phonics instruction is so central to learning in a writing system such as English. But we also move beyond phonics, reviewing research on what else children need to learn to become expert readers and considering how this might be translated into effective classroom practice. We call for an end to the reading wars and recommend an agenda for instruction and research in reading acquisition that is balanced, developmentally informed, and based on a deep understanding of how language and writing systems work.

1 Comment

Filed under Education, Research, Review

A small review of the new book by Andreas Schleicher: World Class

Andreas Schleicher has published a new book called World Class, with as subtitle “How to build a 21st-century school system” and you can download the book for free here.

I’ve been able to read most of the book already and for people who haven’t been following PISA and Schleicher there is a lot of information in the book. For people like me who have been following intensively the international survey and it’s boss, there is not much new stuff to discover.

I could repeat some of the elements from my open letter to Schleicher, although he does try to defend his answer to questions about the importance of memorization since the discussion he had at the Wise conference last year. Again, I’ve seen the charts he uses at page 233 and 234 before, but they don’t explain the chart from PISA 2015 I shared before.

But I want to focus on 2 details that worried me a bit:

  • There is this paragraph on the Flemish Educational system, one that I know very well because I’m living and working here. I do think a lot of Flemish politicians, teachers and researchers would look a bit puzzled when they read this:

The Flemish Community of Belgium benefits from many of the advantages of school choice, such as a wide variety of pedagogies, which offers real choice for parents, and a strong drive towards quality, through competition between schools. It also suffers from some of the disadvantages of school choice, such as a relatively high level of socio-economic segregation among schools and a strong relationship between family background and learning outcomes. But overall, the education system largely succeeds in limiting inequity and social segregation by implementing some steering and accountability mechanisms that apply to all schools.

Well, the first part in which Schleicher mentions the segregation, that has been at the center of our discussions for quite a while now, but the second part? Well, that will be news for a lot of people (and I don’t think they will agree).

  • Secondly there are a lot of sources missing in the book on what we do know about education. A key element of PISA is that we get at best a view of correlations between policy decisions and possible effects, although we do know that we need other kinds of research to establish causal relations. Schleicher sums up again his list of educational myths dispelled by PISA, forgetting that other studies often showed a more nuanced image to say the least (e.g. when discussing class size). Again Schleicher does mention that maybe other people have said something about it, but he forgets to mention who and what they said.

The book has a feeling of an end of era to me. I don’t know of Andreas Schleicher will be retiring soon, but as the book seems to have some elements of a professional memoir, it sure looks that way. I wished that in that case Schleicher would also have responded to some of the fierce criticisms PISA has received the past few years. Now some parts seem to be more of a self-written liber amicorum as someone mentioned earlier to me.

So, do you need to read this book? The advance praise does say so, and I do think it’s a good starting point to read, but while reading you should bear in mind that PISA can be used as a great source, but never should be the only or prime source.

1 Comment

Filed under Research, Review

Important reports to read: metacognition, the word gap and what teacher trainees need to learn

The past few weeks there were so many interesting reports and studies published that it’s sheer impossible to keep track while teaching. In this post I want to highlight what you really shouldn’t miss.

First there is the report that  Daniel Muijs and Christian Bockhove wrote on metacognition and self-regulated learning for the EEF:

This guidance draws on a review of the evidence about self-regulated learning and metacognition led by Professor Daniel Muijs and Dr Christian Bokhove (University of Southampton). It is not a new study in itself, but rather is intended as an accessible overview of existing research with clear, actionable guidance. More information about the review and the process is at the end of the review. Some key references are included here; for those wishing to explore the subject in more depth, the forthcoming report will contain a more comprehensive reference section.

This is a summary of the more important recommendations:

Next up is this Oxford report on why closing the word gap is crucial. It’s a pretty big report, so if you only have limited time, do read the first part with the overview of the evidence, although the whole report is important. You want to know why, well read this excerpt:

Almost all primary school teachers surveyed by OUP felt that the word gap resulted in weaker comprehension skills and slower than expected progress in reading and writing. Secondary school teachers overwhelmingly stated that a low vocabulary affects a child’s progress through secondary school across a wide range of subjects. The vast majority of those surveyed (86% of primary teachers and 80% of secondary teachers) responded that they thought it was very or extremely challenging for pupils with a limited vocabulary to read national test papers. Consequently, 82% primary teachers and 79% of secondary teachers noted that these pupils are likely to get worse results in national tests.

And also:

Self esteem, behaviour and a child’s ability to make friends were all felt to be negatively affected by low levels of vocabulary.

A third report isn’t actually a report but Daniel Willingham who is making a plea about what teacher trainees should learn about educational psychology.  You can check:

This abstract from the article in Mind, Brain, Education summarizes the main idea:

Although most teacher education programs include instruction in the basic science of psychology, practicing teachers report that this preparation has low utility. Researchers have considered what sort of information from psychology about children’s thinking, emotion, and motivation would be useful for teachers’ practice. Here, I take a different tack. I begin by considering three varieties of statements in basic science: empirical observations, theoretical statements, and epistemic assumptions. I suggest that the first of these can support classroom application, but the latter two cannot. I use that conclusion as a starting point for considering the instruction of prospective teachers in psychology.

I for myself am not that sure if I agree with Daniel on this line of thinking, but I do think it’s an important debate to have and therefor well worth the read.

1 Comment

Filed under Education, Research, Review

Believe the scientist or not, it also depends on the quality of the audio

It seems a detail, and actually it is. But the devil is in the details in this case for sure as this new study shows that when people listen to recordings of a scientist presenting their work, the quality of audio has a significant impact on whether people believed what they were hearing. Even worse: this was regardless of who the researcher was or what they were talking about. Blimey!

Little extra information not mentioned in both the press release nor in the abstract:

  • For experiment 1: ninety-seven Amazon Mechanical Turk workers listened and responded to both segments.
  • For experiment 2: Ninety-nine Amazon Mechanical Turk workers listened and responded to both radio segments.

From the press release:

Dr Newman, of the ANU Research School of Psychology, said the results showed when it comes to communicating science, style can triumph over substance.

“When people are assessing the credibility of information, most of the time people are making a judgement based on how something feels,” Dr Newman said.”Our results showed that when the sound quality was poor, the participants thought the researcher wasn’t as intelligent, they didn’t like them as much and found their research less important.”

The study used experiments where people viewed video clips of scientists speaking at conferences. One group of participants heard the recordings in clear high-quality audio, while the other group heard the same recordings with poor-quality audio.

Participants were then asked to evaluate the researchers and their work. Those who listened to the poorer quality audio consistently evaluated the scientists as less intelligent and their research as less important.

In a second experiment, researchers upped the ante and conducted the same experiment using renowned scientists discussing their work on the well-known US Science Friday radio program. This time the recordings included audio of the scientists being introduced with their qualifications and institutional affiliations.”It made no difference,” she said.”As soon as we reduced the audio quality, all of a sudden the scientists and their research lost credibility.”

As with the first experiments, participants thought the research was worse, the scientists were less competent and they also reported finding their work less interesting.

Dr Newman said in a time when genuine science is struggling to be heard above fake news and alternate facts, researchers need to consider not only the content of their messages, but features of the delivery.

“Another recent study showed false information travels six times faster than real information on Twitter,” she said.”Our results show that it’s not just about who you are and what you are saying, it’s about how your work is presented.”

A research paper for the study has been published in the journals Science Communication.

Abstract of the study:

Increasingly, scientific communications are recorded and made available online. While researchers carefully draft the words they use, the quality of the recording is at the mercy of technical staff. Does it make a difference? We presented identical conference talks (Experiment 1) and radio interviews from NPR’s Science Friday (Experiment 2) in high or low audio quality and asked people to evaluate the researcher and the research they presented. Despite identical content, people evaluated the research and researcher less favorably when the audio quality was low, suggesting that audio quality can influence impressions of science.

Leave a comment

Filed under Education, Media literacy, Research, Review

A report of the OECD with a plea for more pedagogy in thinking about education, great but…

There is a new report by the OECD called Teachers as Designers of Learning Environments: the Importance of Innovative Pedagogies, and a Teaching In Focus summary that can be read here.

The bottom line is the following:

Identifying clusters of innovative pedagogies is the first step in developing a broad international consensus of pedagogy across the teaching profession. Such a framework needs to start with the argument that teachers are high-level professionals whose professionalism revolves around collaborative pedagogical expertise. To call for a pedagogical framework is to recognise the key role of pedagogy, not to ask policies to dictate the best teaching methods. It is a matter of widening the skills of teachers to promote more interactive, horizontal and caring relationships with students. In focusing on the role of teachers as creative professionals, a framework for pedagogies calls for a form of teaching that retains a deliberate form of lesson planning that promotes student centredness and active participation.

Finally, by starting to think about the relationships of teaching and learning around natural learning inclinations, such as play, creativity, collaboration, and inquiry, the clusters of innovative pedagogies consciously promote the engagement of learners and match the fundamentals of learning to improve the professional competences of teachers.

I know there is often a confusion between didactics and pedagogy, something the authors of the report also are a bit blurry about, but in this report there is another mistake being made imho. This report is not a plea for more pedagogy in education – which I think would have been correct – but for more of a certain vision of pedagogy in education. If you read the recent work by Biesta or Furedi, one could think that present discussion in thinking about pedagogy are already a bit further than the Rousseau-influence one still can taste in this report (I wrote an academic paper on this new line of thinking, but sadly enough I wrote it in Dutch). Recently Simons and Masschelein wrote a book (again in Dutch, I did try to convince my publisher to translate the book into English) in which they looked at what student- or pupil-centered actually means and they end up with much more questions.

But there is more. Again in my personal opinion, I do think that pedagogy and pedagogical innovations shouldn’t be blind for insights from didactical – evidence-informed – and cognitive psychological or epistemological research – the distinction between primary and secondary learning processes – , or even research by the OECD themselves showing that inquiry based learning does have issues.

I know there seems to be a polarization – again – in thinking about education, but everybody who thinks that it is centered around two poles, is making a mistake. E.g. an author such as Biesta has questioned both PISA and the OECD as evidence-based education that has been adapted by people who are at least regarded by some as more conservative. I do think there is an interesting but broad field of thinking about education, and while I’m certainly pro evidence-informed, I also see the need for relationship and subscribe the importance of thinking about the why of education.  That’s why I do think this report is too one-dimensional in its approach. Btw, I also read the working paper related to this report and blimey couldn’t find a Hattie, Biesta, or other Furedis being mentioned at all. They do refer to a lot of work on ICT in education.

1 Comment

Filed under Education, Review