Category Archives: Review

Does successfully improving students’ achievement test scores lead to higher rates of national economic growth?

No, this isn’t a OECD or PISA-bashing post, but I found a new study by Komatsua and Rappleye via @cbokhove that raises an important question: does successfully improving students’ achievement test scores lead to higher rates of national economic growth. This is a claim based on research by Hanushek and Woessmann and is the basis for a lot of policy-influencing research and policy-advice by e.g. PISA or the World bank. But Komatsua and Rappleye argue now that this claim is maybe based on flawed statistics, as the abstract makes clear what it’s all about:

Several recent, highly influential comparative studies have made strong statistical claims that improvements on global learning assessments such as PISA will lead to higher GDP growth rates. These claims have provided the primary source of legitimation for policy reforms championed by leading international organisations, most notably the World Bank and OECD. To date there have been several critiques but these have been too limited to challenge the validity of the claims. The consequence is continued utilisation and citation of these strong claims, resulting in a growing aura of scientific truth and concrete policy reforms. In this piece we report findings from two original studies that invalidate these statistical claims. Our intent is to contribute to a more rigorous global discussion on education policy, as well as call attention to the fact that the new global policy regime is founded on flawed statistics.

They performed a replication on the same data Hanushek and Woessmann used and their conclusion sounds a bit damning:

Our primary purpose has been to report findings from two studies where our results invalidate H&W’s strong statistical claims that attempt to link student test scores and economic growth. In Study 1, we observed that the explanatory power of test scores was weak in subsequent periods: the relationship between scores and growth has been neither consistently strong nor strongly consistent. In Study 2, we observed that the relationship between changes in test scores in one period and changes in economic growth for subsequent periods were unclear at best, doubtful at worst. Combined, these two original studies do not simply challenge the key statistical claims advanced by H&W but invalidate them because they utilise the same sample, dataset and methods.

But I have to agree with Christian Bokhove in our tweet conversation about this article that both scientists are quite a bit firm in their statements. Still: I think it’s too important not to have further debate on this topic as it’s quite essential to present policy.

1 Comment

Filed under Education, Research, Review

Important new meta-analysis on the testing effect – with some surprises…

There is a new Best Evidence in Brief and one of the studies the newsletter discusses is all about the effect of testing:

Olusola O. Adesope and colleagues conducted a meta-analysis to summarize the learning benefits of taking a practice test versus other forms of non-testing learning conditions, such as re-studying, practice, filler activities, or no presentation of the material.

Results from 272 independent effects from 188 separate experiments demonstrated that the use of practice tests is associated with a moderate, statistically significant weighted mean effect size compared to re-studying (+0.51) and a much larger weighted mean effect size (+ 0.93) when compared to filler or no activities.

In addition, the format, number, and frequency of practice tests make a difference for the learning benefits on a final test. Practice tests with a multiple-choice option have a larger weighted mean effect size (+0.70) than short-answer tests (+0.48). A single practice test prior to the final test is more effective than when students take several practice tests. However, the timing should be carefully considered. A gap of less than a day between the practice and final tests showed a smaller weighted effect size than when there is a gap of one to six days (+0.56 and + 0.82, respectively).  

Are you as baffled by these results as I am? I checked in the original article to find out more.

Eg. MC-questions having a bigger effect size – while remembering is often harder than recognizing? Well, there are some studies who actually support the latter:

For instance, Kang et al. (2007) revealed that students who took a short-answer practice test outperformed students who took a multiple-choice practice test on the final test, regardless of whether the final test was short-answer or multiple-choice.

But:

On the other hand, C. D. Morris, Bransford, and Franks’s (1977) research on levels of processing suggests that retention is strongest when processing demands are less demanding. They reason that this is because less demanding retrieval practice activities allow participants to focus all of their cognitive energy on a simple task at hand, whereas deeper levels of processing require more cognitive energy and can distract participants from relevant aspects (C. D. Morris et al., 1977).

And looking at the meta-analysis, the second theory seems to be winning as “the differences between multiple-choice and short-answer practice test formats did emerge: g = 0.70 and g = 0.48, respectively” But it’s worth noting that the researchers do warn it’s not that simple:

…we found that multiple-choice testing was the most effective format; however, this should be interpreted with caution, since an educator’s decision to use any given format should be based on the content of the learning material and the expected learning outcomes. For example, multiple- choice tests may be especially useful for memorization and fact retention, while short-answer testing may require more higher order thinking skills that are useful for more conceptual and abstract learning content

And what about a single test being more effective than taking several practice tests? The meta-analysis does support this, but Adesope et al. can only guess why:

Thus, our findings suggest that although a single test prior to a final test may result in better performance, the timing of the test should be carefully considered. One plausible explanation is more time between the practice and final tests allows students to mentally recall and process information, leading to deeper learning. An alternative hypothesis is that multiple tests within a short time may result in test fatigue that affects performance, while retrieval practice over a distributed time period enables long-term storage.

I do think that this meta-analysis will invite other researcher do join the debate…

Abstract of the study:

The testing effect is a well-known concept referring to gains in learning and retention that can occur when students take a practice test on studied material before taking a final test on the same material. Research demonstrates that students who take practice tests often outperform students in nontesting learning conditions such as restudying, practice, filler activities, or no presentation of the material. However, evidence-based meta-analysis is needed to develop a comprehensive understanding of the conditions under which practice tests enhance or inhibit learning. This meta-analysis fills this gap by examining the effects of practice tests versus nontesting learning conditions. Results reveal that practice tests are more beneficial for learning than restudying and all other comparison conditions. Mean effect sizes were moderated by the features of practice tests, participant and study characteristics, outcome constructs, and methodological features of the studies. Findings may guide the use of practice tests to advance student learning, and inform students, teachers, researchers, and policymakers. This article concludes with the theoretical and practical implications of the meta-analysis.

1 Comment

Filed under Education, Research, Review

New website for Evidence-based education: Evidence for ESSA

Evidence for ESSA, is a free web-based resource that provides easy access to information on programs that meet the evidence standards defined in the American Every Student Succeeds Act (ESSA). The website is designed for education leaders at the state, district, and school levels, to provide information to state chiefs, superintendents, principals, teachers, parents, and anyone else interested in which programs meet the ESSA evidence standards.

While this website is aimed at educators in the United States, I do think it can be interesting to a global audience as the website reviews math and reading programs for grades K to 12 to determine which meet the strong, moderate, or promising levels of evidence defined in ESSA (additional subject areas will be added later). The site provides a one-page summary of each program, including a program description, brief research review, and practical information on costs, professional development, and technology requirements. It is easily searchable and searches can be refined for particular groups (such as African Americans, Hispanics, or English Learners); communities (urban, suburban, or rural); and program features (such as technology, cooperative learning, or tutoring). Evidence for ESSA directs users to the key studies that validate that a program meets a particular ESSA standard. 

The mission of Evidence for ESSA is to provide clear and authoritative information on programs that meet the ESSA evidence standards and enable educators and communities to select effective educational tools to improve student success.

1 Comment

Filed under Education, Research, Review

Infographic by Oliver Caviglioli and Paul Kirschner: the case for fully-guided instruction

Oliver Caviglioli The same guy who designed the great study method posters by the learning scientists worked now together with Paul Kirschner on this infographic which you can download here!

3 Comments

Filed under Education, Review

Interesting read: What Works Can Hurt: Side Effects in Education

I’ve been working on a new book, due out January 2018, about teaching and my first chapter – already read by my publishers – has quite some in common with this new blog post and article by Yong Zhao: What Works Can Hurt: Side Effects in Education.

This is the abstract of the actual paper (you can download the actual article also in this blog post):

Medical research is held as a field for education to emulate. Education researchers have been urged to adopt randomized controlled trials, a more ‘‘scien- tific’’ research method believed to have resulted in the advances in medicine. But a much more important lesson education needs to borrow from medicine has been ignored. That is the study of side effects. Medical research is required to investigate both the intended effects of any medical interventions and their unintended adverse effects, or side effects. In contrast, educational research tends to focus only on proving the effectiveness of practices and policies in pursuit of ‘‘what works.’’ It has generally ignored the potential harms that can result from what works. This article presents evidence that shows side effects are inseparable from effects. Both are the outcomes of the same intervention. This article further argues that studying and reporting side effects as part of studying effects will help advance education by settling long fought battles over practices and policies and move beyond the vicious cycle of pendulum swings in education.

2 Comments

Filed under Education, Review

Interesting read: Why global education rankings don’t reveal the whole picture

What if… you took socio-economic backgrounds of countries into account when you make rankings of educational systems?

This is exactly what Daniel Caro and Jenny Lenkeit did, you can read all about it here.

But the rankings sure look different:

1 Comment

Filed under Education, Review

Best evidence in brief: What works for bullying prevention?

There is a new Best Evidence in Brief and I’m sure many teachers and parents alike want to read this study:

Child Trends has released a new policy brief on preventing bullying and cyberbullying. The report provides information on the current state of bullying research using data from the U.S Department of Education, journal articles, and existing research by Child Trends, and provides recommendations for addressing and preventing bullying behavior.

The report notes that while many bullying prevention programs and strategies are available, evidence of their effectiveness has been mixed, and most have never been rigorously evaluated. Based on the existing research, the report provides the following recommendations:

  • Include cyberbullying as part of a broader approach to bullying prevention. Strategies targeting cyberbullying alone without addressing the broader issue of bullying are unlikely to be effective. Similarly, monitoring students’ social media accounts is likely to be an ineffective use of resources without additional efforts to encourage more civil behavior online and in person.
  • Support the development of evidence-based approaches through dedicated funding for research. Such investments should also examine interventions, such as integrated student supports, for students who are targeted by bullying or witness it.
  • Discourage approaches that lack evidentiary support, criminalize youth, or remove youth from school. Research shows that anti-bullying assemblies, speakers, and campaigns are not effective at preventing bullying, nor are zero-tolerance policies that remove students from school and do not address the underlying causes of bullying behavior.

1 Comment

Filed under Education, Research, Review

What I learned about autonomy from Richard Ryan (cfr Deci & Ryan) yesterday

Yesterday I had the pleasure to attend a public lecture by Richard Ryan at Ghent University. Ryan is famous for his work with Deci on the Self Determination Theory (SDT).

For the people who don’t know this theory, a short reminder:

Yesterday Ryan gave an overview, but when he discussed Autonomy he suddenly said something that I thought was really important for education:

So: SDT is no excuse for e.g. letting children learn without guidance or structure. Ryan made it very clear later on in his talk that autonomy within structured classes was the most effective – he also mentioned scaffolding in this case.

I did notice that most of the time Ryan mentioned links between this motivation-theory and wellbeing. He did mention a link between performance for bankers and the theory (but on the slide you could see this correlation was much smaller than on feelings of wellbeing) and the only time the effect on learning was mentioned, was in what I discussed above.

He did also say this, which is a very nice quote:

1 Comment

Filed under Education, Psychology, Research, Review

My keynote presentation for researchED Haninge #rEDHan

2 Comments

Filed under Education, Review

Interesting read: A Mindset “Revolution” Sweeping Britain’s Classrooms May Be Based On Shaky Science

Growth and fixed mindset are all the rage, but this article by Tom Chivers shows something else: the research behind it contains worrying errors.

An excerpt from the article:

But the striking effects in Dweck’s findings have surprised psychologists. Timothy Bates, a professor of psychology at the University of Edinburgh, told BuzzFeed News that the “big effects, monstrous effects” that Dweck has found in the 1998 study and others are “strange – it’s an odd one to me”.

Bates told BuzzFeed News that he has been trying to replicate Dweck’s findings in that key mindset study for several years. “We’re running a third study in China now,” he said. “With 200 12-year-olds. And the results are just null.

“People with a growth mindset don’t cope any better with failure. If we give them the mindset intervention, it doesn’t make them behave better. Kids with the growth mindset aren’t getting better grades, either before or after our intervention study

Dweck told BuzzFeed News that attempts to replicate can fail because the scientists haven’t created the right conditions. “Not anyone can do a replication,” she said. “We put so much thought into creating an environment; we spend hours and days on each question, on creating a context in which the phenomenon could plausibly emerge.

“Replication is very important, but they have to be genuine replications and thoughtful replications done by skilled people. Very few studies will replicate done by an amateur in a willy-nilly way.”

Nick Brown, a PhD student in psychology at the University of Groningen in the Netherlands, is sceptical of this: “The question I have is: If your effect is so fragile that it can only be reproduced [under strictly controlled conditions], then why do you think it can be reproduced by schoolteachers?”

And Nick Brown did much more, you can read it here.

6 Comments

Filed under Education, Myths, Research, Review