A week in replication, some failed, some successful: Big Five, note taking and stereotype threats

The past week I read several replication studies with mixed results:

• The good news, most research on personality, more specific The Big Five, replicates! Check the pre-print or this BPS Digest.
• Is the pen mightier than the keyboard for taking notes. It is a study that I’ve mentioned quite a lot before, but the replication now seemingly failed. But it’s a bit more complicated:

First, although direct replications (using methods from Mueller and Oppenheimer 2014) yielded longhand-superiority effects, the effects relevant to the encoding function of note-taking were small and did not reach conventional levels of significance. Such outcomes do not support strong recommendations about whether students should take notes longhand or by laptop in class. Second, based on their evidence, Mueller and Oppenheimer (2014) hypothesized that the higher word count and verbatim overlap for laptop groups were in part responsible for diminished performance for laptop users. However, when combining results across the direct replications, differences in word count and verbatim overlap were large, whereas differences in performance were small (and not statistically significant). Thus, differences in word count and verbatim overlap do not appear sufficient to produce performance differences between longhand and laptop groups.

• And than there is the idea that stereotype threats can have an effect of how kids perform on mathematical tests, making girls perform worse. This has often been established through research, but this Dutch high powered, preregistered reported study fails to replicate this.

The effects of gender stereotype threat on mathematical test performance in the classroom have been extensively studied in several cultural contexts. Theory predicts that stereotype threat lowers girls’ performance on mathematics tests, while leaving boys’ math performance unaffected. We conducted a large-scale stereotype threat experiment in Dutch high schools (N = 2064) to study the generalizability of the effect. In this registered report, we set out to replicate the overall effect among female high school students and to study four core theoretical moderators, namely domain identification, gender identification, math anxiety, and test difficulty. Among the girls, we found neither an overall effect of stereotype threat on math performance, nor any moderated stereotype threat effects. Most variance in math performance was explained by gender, domain identification, and math identification. We discuss several theoretical and statistical explanations for these findings. Our results are limited to the studied population (i.e. Dutch high school students, age 13–14) and the studied domain (mathematics).

In both examples of failed replication it’s not the case that we should abolish the theories as such straightaway. It’s rather that these new studies show that it’s more complicated than thought maybe due to regional and age differences (stereotype threat), maybe due to maybe differences in personal experiences (note taking).