People who often read this blog know that one of the reasons I write posts is to debunk myths, mostly in education. Quite often these are neuromyths, because neurology is getting quite popular in education.
This study published in Nature by Button et al. is a whole different story. The analysis doesn’t attack neuromyths, but the neurological research itself and the team that has done this consists of neuroscientists, psychologists, geneticists and statisticians. They analysed meta-analyses of neuroscience research to determine the statistical power of the papers contained within and found that there is something wrong with the power. But what is power?
The power of a statistical test is the probability that the test will reject the null hypothesis when the null hypothesis is false (i.e. the probability of not committing a Type II error, hence the probability of confirming the alternative hypothesis when the alternative hypothesis is true). The power is in general a function of the possible distributions, often determined by a parameter, under the alternative hypothesis. As the power increases, the chances of a Type II error occurring decrease. The probability of a Type II error occurring is referred to as the false negative rate (β). Therefore power is equal to 1 − β, which is also known as the sensitivity.
Power analysis can be used to calculate the minimum sample size required so that one can be reasonably likely to detect an effect of a given size. Power analysis can also be used to calculate the minimum effect size that is likely to be detected in a study using a given sample size. In addition, the concept of power is used to make comparisons between different statistical testing procedures: for example, between a parametric and a nonparametric test of the same hypothesis.
What happens if research is underpowered? Well, with too few samples there is a big risk for false positives, something that might be happening in neuroscience.
Abstract of the research:
A study with low statistical power has a reduced chance of detecting a true effect, but it is less well appreciated that low power also reduces the likelihood that a statistically significant result reflects a true effect. Here, we show that the average statistical power of studies in the neurosciences is very low. The consequences of this include overestimates of effect size and low reproducibility of results. There are also ethical dimensions to this problem, as unreliable research is inefficient and wasteful. Improving reproducibility in neuroscience is a key priority and requires attention to well-established but often ignored methodological principles.