Found this via Tim Surma (do follow him!), check the source here.
Category Archives: Review
A week in replication, some failed, some successful: Big Five, note taking and stereotype threats
The past week I read several replication studies with mixed results:
- The good news, most research on personality, more specific The Big Five, replicates! Check the pre-print or this BPS Digest.
- Is the pen mightier than the keyboard for taking notes. It is a study that I’ve mentioned quite a lot before, but the replication now seemingly failed. But it’s a bit more complicated:
First, although direct replications (using methods from Mueller and Oppenheimer 2014) yielded longhand-superiority effects, the effects relevant to the encoding function of note-taking were small and did not reach conventional levels of significance. Such outcomes do not support strong recommendations about whether students should take notes longhand or by laptop in class. Second, based on their evidence, Mueller and Oppenheimer (2014) hypothesized that the higher word count and verbatim overlap for laptop groups were in part responsible for diminished performance for laptop users. However, when combining results across the direct replications, differences in word count and verbatim overlap were large, whereas differences in performance were small (and not statistically significant). Thus, differences in word count and verbatim overlap do not appear sufficient to produce performance differences between longhand and laptop groups.
- And than there is the idea that stereotype threats can have an effect of how kids perform on mathematical tests, making girls perform worse. This has often been established through research, but this Dutch high powered, preregistered reported study fails to replicate this.
The effects of gender stereotype threat on mathematical test performance in the classroom have been extensively studied in several cultural contexts. Theory predicts that stereotype threat lowers girls’ performance on mathematics tests, while leaving boys’ math performance unaffected. We conducted a large-scale stereotype threat experiment in Dutch high schools (N = 2064) to study the generalizability of the effect. In this registered report, we set out to replicate the overall effect among female high school students and to study four core theoretical moderators, namely domain identification, gender identification, math anxiety, and test difficulty. Among the girls, we found neither an overall effect of stereotype threat on math performance, nor any moderated stereotype threat effects. Most variance in math performance was explained by gender, domain identification, and math identification. We discuss several theoretical and statistical explanations for these findings. Our results are limited to the studied population (i.e. Dutch high school students, age 13–14) and the studied domain (mathematics).
In both examples of failed replication it’s not the case that we should abolish the theories as such straightaway. It’s rather that these new studies show that it’s more complicated than thought maybe due to regional and age differences (stereotype threat), maybe due to maybe differences in personal experiences (note taking).
Filed under Education, Psychology, Research, Review
Can you train executive functions? A second meta-analysis says…
In 2015 I wrote this blogpost on a meta-analysis on the training of executive functions. The results were a bit depressing. Today I found a new meta-analysis via Jeroen Jansen and the results… are again not that good and it all has to do with one of the key concepts in education: transfer. The meta-analysis was based on 38 studies with 47 contrasts.
I’ll first share with you the abstract of the study:
In the present meta-analysis we examined the near- and far-transfer effects of training components of children’s executive functions skills: working memory, inhibitory control, and cognitive flexibility. We found a significant near-transfer effect (g+ = 0.44, k = 43, p < .001) showing that the interventions in the primary studies were successful in training the targeted components. However, we found no convincing evidence of far-transfer (g+ = 0.11, k = 17, p = .11). That is, training a component did not have a significant effect on the untrained components. By showing the absence of benefits that generalize beyond the trained components, we question the practical relevance of training specific executive function skills in isolation. Furthermore, the present results might explain the absence of far-transfer effects of working memory training on academic skills (Melby-Lervag & Hulme, 2013; Sala & Gobet, 2017).
So, this study is not that surprising as it states that near transfer can happen more easily, far transfer is far more difficult (pun intended). But the consequences of this finding can be pretty big:
The most important aim of the present meta-analysis was to gain a deeper understanding of whether and to what extent the different components of executive functions are trainable and whether train- ing a specific executive function has an ameliorating effect on other main executive function components. This is of crucial importance because in the long run the main aim of training executive functions skills is to improve children’s everyday functioning; for example, academic and social skills as well as emotion regulation. These complex skills are not supported by one sole executive function but generally rely on the interplay among most of them.
So to be clear the study doesn’t say that executive functions can’t be trained, or…
Overall, we found a significant, medium-sized near-transfer effect. However, no far-transfer effect appeared. More specifically, there were significant near-transfer effects on all three components: a moderate-sized effect on working memory and small-sized effects on inhibition and cognitive flexibility. In contrast, no far-transfer effects were found on working memory, inhibitory control, or flexibility. The finding that there was a significant near-transfer effect excludes the possibility that the interventions in the primary studies were not effective in training the components that they targeted. Instead, performance on the components that were trained did significantly improve, however, these gains did not transfer to the untrained components.
But what does this mean?
The lack of far-transfer effect found in the present meta-analysis even within the set of executive function skills makes it—though logically not impossible—still highly unlikely that training unique executive functions could have measurable ameliorating transfer effect on more distantly related and complex constructs, such as academic and social skills (Blair & Razza, 2007) that rely just as much on the trained executive function component as on the other untrained and largely unaffected components. The results of the present meta-analysis therefore provide a possible explanation for the previously found absence of far-transfer effects of working memory trainings on academic skills.
So what do we need to do?
The present meta-analysis shows that there are limited practical benefits— other than on the trained component— of training single executive function components in childhood. Thus, it might be more advisable, both in the educational and in the clinical practice, to use approaches that target multiple executive function components.
But maybe the most important question is in the end of the meta-analysis:
“The lack of causal evidence for significant relationships among the three core components might contradict accounts of executive functions as a single construct.”
So maybe you can’t train ‘executive functions‘, but you can train an executive function.
A meta-analysis on computer-supported collaborative learning (Best Evidence in Brief)
There is a new Best Evidence in Brief with among others, this study that might surprise some people and oh, my dear friend and co-author Paul Kirschner was involved:
Juanjuan Chen and colleagues recently performed a meta-analysis on the effects of computer-supported collaborative learning (CSCL).
Using 425 empirical studies (all of which used a controlled experimental or quasi-experimental design) published between 2000 and 2016, researchers found several main characteristics to examine: the effects of the collaboration itself; the effects of computer use during collaboration; the effects of extra technology-related learning tools used in CSCL, such as videoconferencing and sharing visuals with team partners; and strategies such as role assignment and peer feedback.Collaborative learning itself positively affected:
- Knowledge gain (+0.42)
- Skill acquisition (+0.62)
- Student perceptions of the experience (+0.38)
The use of computers, when combined with collaborative learning, positively affected:
- Knowledge gain (+0.45)
- Skill acquisition (+0.53)
- Student perceptions (+0.51)
- Group task performance (+0.89)
- Social interaction (+0.57)
Lastly, extra technology-related learning tools during CSCL positively affected knowledge gain (+0.55), as did the use of strategies (+0.38).
What’s really new in the “Innovating Pedagogy 2019”-report?
The title suggests one of my dominant feelings when reading the Innovating Pedagogy 2019-report by the Open University. The report discusses 9 topics, and many of them aren’t really new to me. Or do you think that ‘Playful learning’, ‘Learning through wonder’, ‘making thinking visible’ or ‘Place-based learning’ are so new. Huizinga died in 1945 but already wrote about the importance of play, inspiring thinking about playful learning for decades, learning through wonder is nothing more than this:
something any good teacher already knows, making thinking visible is closely related to dual channel or dual coding theory and place-based learning? Well, field-trips are so new…
Other elements discussed in the report are more related to the present peak of inflated expectations of the technology hype cycle, e.g. teaching with robots (do note that the LOGO-turtle celebrated it’s fiftieth birthday 2 years ago), teaching with drones and virtual studios.
Maybe the most current element in the report, is probably also the most controversial for some: Decolonising learning, which is described as:
A curriculum provides a way of identifying the knowledge we value. It structures the ways in which we are taught to think and talk about the world. As education has become increasingly global, communities have challenged the widespread assumption that the most valuable knowledge and the most valuable ways of teaching and learning come from a single European tradition. Decolonising learning prompts us to consider everything we study from new perspectives. It draws attention to how often the only world view presented to learners is male, white, and European.
This does give food for thought, although it’s part of a bigger discussion. Closely related is the last element discusses: ‘Roots of empathy’.
Roots of Empathy is a classroom programme that is designed to teach children empathy. It prepares children aged 5 to 13 to interact with others healthily and constructively. It also prepares them to cope with different relationships in their lives. This programme is based on the principle that when children understand how they feel and how other people feel, they find it easier to cope in social situations. In order to help them to do this, Roots of Empathy develops their emotional understanding. Evaluations of the approach show that it decreases children’s aggressive behaviour, improves social behaviour, and, due to its emphasis on the actions and feelings of babies, increases the knowledge children have about infant development.
Both elements are actually closest related to the continental thinking about pedagogy.
About nature versus nurture: the four laws of behavioural genetics
This tweet by Steve Stewart-Williams is so relevant I wanted to share it here on this blog as I know a lot of people who follow my posts aren’t on Twitter.
If you feel angry after reading the first two laws, do read on. Both articles mentioned in the tweet are also must reads.
Filed under Psychology, Research, Review
A lecture by and a conversation with Steven Pinker: why we should be happier with the world today
Filed under Media literacy, Myths, Research, Review, Trends
A new review study about Collaborative Problem Solving, still a lot to research
I found this new review study via Jeroen Janssen and it’s quite interesting as it tackles Collaborative Problem Solving and how much is already known about how to advance this. Spoiler: not that much, it seems, while it now is being measured e.g. by PISA.
But first what is it? That is already a bit more difficult question to answer, but I do think this description of what is needed makes it more clear:
…team members must be able to define the problem, understand who knows what on the team, identify gaps in what is known and what is required, integrate these to generate candidate solutions, and monitor progress in achieving the group goals. From the social perspective, the success of a team requires that members establish shared understanding, pursue joint and complementary actions, and coordinate their behavior in service of generating and evaluating solutions.
So what is (un)known? This overview at the end of the article with suggestions for further research, makes a lot more clear:
- CPS has been identified as an important skill in the international community and workforce, but recent assessments have revealed that students and adults have low CPS proficiency. This calls for an analysis of CPS mechanisms, frequent problems, and methods of solving these problems. Psychological scientists could play a major role in this broad effort by partnering with stakeholders.
- CPS is rarely trained in schools and the workforce, and the existing training is not informed by psychological science. This opens the door to the value of psychological scientists’ being part of national and international efforts drawing from their expertise in science, learning and training.
- Psychological scientists have developed a body of empirical research and theory of team science over the years, but much of this work has focused on group learning, work, memory, and decision making rather than CPS per se. We need to sort out how much of the existing research in team science applies to CPS. Psychological scientists are encouraged to direct their focus on CPS per se in the team science research landscape.
- Intelligent digital technologies have the potential to automatically analyze large samples of group interactions at multiple levels of language, discourse, and interactivity. This is landmark progress because existing research on teams has had small samples and time-consuming annotation of the interactions. There is a need for psychological scientists to partner with the developers of these technologies to recommend psychological characteristics to track and to scrutinize the validity of automated measures.
- A curriculum for training CPS competencies has not been developed and adequately tested. There is a need to develop a program of research on CPS curriculum design for both students and instructors. Psychological scientists are an important asset to generate potential curricula and to test their efficacies.
Abstract of the open access review:
Collaborative problem solving (CPS) has been receiving increasing international attention because much of the complex work in the modern world is performed by teams. However, systematic education and training on CPS is lacking for those entering and participating in the workforce. In 2015, the Programme for International Student Assessment (PISA), a global test of educational progress, documented the low levels of proficiency in CPS. This result not only underscores a significant societal need but also presents an important opportunity for psychological scientists to develop, adopt, and implement theory and empirical research on CPS and to work with educators and policy experts to improve training in CPS. This article offers some directions for psychological science to participate in the growing attention to CPS throughout the world. First, it identifies the existing theoretical frameworks and empirical research that focus on CPS. Second, it provides examples of how recent technologies can automate analyses of CPS processes and assessments so that substantially larger data sets can be analyzed and so students can receive immediate feedback on their CPS performance. Third, it identifies some challenges, debates, and uncertainties in creating an infrastructure for research, education, and training in CPS. CPS education and assessment are expected to improve when supported by larger data sets and theoretical frameworks that are informed by psychological science. This will require interdisciplinary efforts that include expertise in psychological science, education, assessment, intelligent digital technologies, and policy.
Very interesting and relevant talk by Robert A. Bjork on learning, memory and forgetting (and how bad we are in judging effective learning)
H/T to @triciatailored
What do you need to succeed in life?
The answer is of course sheer luck, besides talent and intelligence. This new systematically review doesn’t say intelligence and talent aren’t needed, but suggests that non-cognitive skills can also be important, although there are also some serious warning lights surrounding the existing body of evidence.
From the press release:
The study, published in the journal Nature Human Behaviour is the first to systematically review the entire literature on effects of non-cognitive skills in children aged 12 or under, on later outcomes in their lives such as academic achievement, and cognitive and language ability.
“Traits such as attention, self-regulation, and perseverance in childhood have been investigated by psychologists, economists, and epidemiologists, and some have been shown to influence later life outcomes,” says Professor John Lynch, School of Public Health, University of Adelaide and senior author of the study.
“There is a wide range of existing evidence under-pinning the role of non-cognitive skills and how they affect success in later life but it’s far from consistent,” he says.
One of the study’s co-authors, Associate Professor Lisa Smithers, School of Public Health, University of Adelaide says: “There is tentative evidence from published studies that non-cognitive skills are associated with academic achievement, psychosocial, and cognitive and language outcomes, but cognitive skills are still important.”
One of the strongest findings of their systematic review was that the quality of evidence in this field is lower than desirable. Of over 550 eligible studies, only about 40% were judged to be of sufficient quality.
“So, while interventions to build non-cognitive skills may be important, particularly for disadvantaged children, the existing evidence base underpinning this field has the potential for publication bias and needs to have larger studies that are more rigorously designed. That has important implications for researchers and funding agencies who wish to study effects of non-cognitive skills,” says Professor Lynch.
Abstract of the study:
Success in school and the labour market relies on more than high intelligence. Associations between ‘non-cognitive’ skills in childhood, such as attention, self-regulation and perseverance, and later outcomes have been widely investigated. In a systematic review of this literature, we screened 9,553 publications, reviewed 554 eligible publications and interpreted results from 222 better-quality publications. Better-quality publications comprised randomized experimental and quasi-experimental intervention studies (EQIs) and observational studies that made reasonable attempts to control confounding. For academic achievement outcomes, there were 26 EQI publications but only 14 were available for meta-analysis, with effects ranging from 0.16 to 0.37 s.d. However, within subdomains, effects were heterogeneous. The 95% prediction interval for literacy was consistent with negative, null and positive effects (−0.13 to 0.79). Similarly, heterogeneous findings were observed for psychosocial, cognitive and language, and health outcomes. Funnel plots of EQIs and observational studies showed asymmetric distributions and potential for small study bias. There is some evidence that non-cognitive skills associate with improved outcomes. However, there is potential for small study and publication bias that may overestimate true effects, and the heterogeneity of effect estimates spanned negative, null and positive effects. The quality of evidence from EQIs underpinning this field is lower than optimal and more than one-third of observational studies made little or no attempt to control confounding. Interventions designed to develop children’s non-cognitive skills could potentially improve outcomes. The interdisciplinary researchers interested in these skills should take a more strategic and rigorous approach to determine which interventions are most effective.
Filed under Education, Psychology, Research, Review
