Replication is a hot topic in research – although maybe not enough in educational research – and there are clear proponents but there are also scientists who talk about the ‘replication police’. A new study by Bench et al. sheds an interesting light on the results of replication studies and sees a link between successful replications and the expertise of the researchers doing the replication:
- High expertise teams obtained effect sizes larger than low expertise teams.
- High expertise teams also selected original studies with larger effect sizes.
- The overall pattern of results suggested expertise mostly impacts study selection.
- Experts may consider different methodological criteria at study selection.
Also interesting to note:
High expertise teams also selected studies with smaller sample sizes, but did not differ from low expertise teams on selection of studies based onp-value or the counterintuitiveness of the finding, suggesting that this selection difference was not driven by a desire to replicate (or avoid replicating) studies with red flags.
The authors of the study think that experienced researchers are “…more easily able to deduce which studies were robust and generalizable.”
Also important to note is that the sample in this study is rather small. They examined the original 100 studies that were included in the OSC (2015) report, minus the three studies that the authors of this report excluded.
Abstract of the study:
A recent article reported difficulty in replicating psychological findings and that training and other moderators were relatively unimportant in predicting replication effect sizes. Using an objective measure of research expertise (number of publications), we found that expertise predicted larger replication effect sizes. The effect sizes selected and obtained by high-expertise replication teams was nearly twice as large as that obtained by low-expertise teams, particularly in replications of social psychology effects. Surprisingly, this effect seemed to be explained by experts choosing studies to replicate that had larger original effect sizes. There was little evidence that expertise predicted avoiding red flags (i.e. the troubling trio) or studies that varied in execution difficulty. However, experts did choose studies that were less context sensitive. Our results suggest that experts achieve greater replication success, in part, because they choose more robust and generalizable studies to replicate.