This is a strange study. Researchers from the University of California in Berkeley used the data of 196000 Lumosity-users. This is a big group, making the study already interesting, but hold your horses, I do think there are some major issues.
First read this excerpt from the press release and see if you can spot the mistake:
The study, led by University of California, Berkeley, researchers, examined relationships between educational attainment, cognitive performance and learning in order to quantify the cumulative effect of attending school.
Its findings suggest that higher levels of education may help stave off age-related cognitive decline. In addition, the team found that education didn’t have a large impact on novel learning, or learning something new at various points in time.
The work, which reviewed the performance of around 196,000 subscribers to Lumosity online brain-training games, is believed to be the largest to date to evaluate cognitive effects of prior educational experience on past and future performance. Researchers said their findings may be of value to psychologists, sociologists, neuroscientists, education researchers and policymakers.
Grading educational achievement
Conventional wisdom has long accepted that higher education is likely to boost incomes and helps prepare individuals for a workplace with often-changing skill sets. Yet fewer than 40 percent of adults in the United States are expected to graduate from college in their lifetimes, and the percentage declines for more advanced degrees.
Until now, research has been inconclusive about the cognitive impacts of higher education and whether the quantity of schooling can influence the acquisition and maintenance of cognitive skills over time.
The researchers of the paper, which appears in the August 23 edition of PLOS ONE, are Silvia Bunge, a professor of psychology at UC Berkeley professor and at the Helen Wills Neuroscience Institute; Belén Guerra-Carrillo, a graduate student in Bunge’s Building Blocks of Cognition Laboratory and a National Science Foundation Fellow; and Kiefer Katovich, who was a statistician with Lumos Labs while the study was conducted.
Bunge and her team say higher levels of education are strong predictors of better cognitive performance across the 15- to 60-year-old age range of their study participants, and appear to boost performance more in areas such as reasoning than in terms of processing speed.
The study’s findings are consistent with prior evidence that the brain adapts in response to challenges, a phenomenon called “experience-dependent brain plasticity.” Based on the principles of plasticity, the authors predicted improvements in cognitive skills that are repeatedly taxed in demanding, cognitively engaging coursework.
Differences in performance were small for test subjects with a bachelor’s degree compared to those with a high school diploma, and moderate for those with doctorates compared to those with only some high school education.
The researchers noted that people from lower educational backgrounds learned novel tasks nearly as well as those from higher ones.
“The fact that the cognitive tests were not similar to what is learned in school is a strength of the study: It speaks to the idea that schooling doesn’t merely impart knowledge – it also provides the opportunity to sharpen core cognitive skills,” said Bunge.
The researchers analyzed anonymized data collected from around 196,000 Lumosity subscribers in the United States, Canada and Australia who came from a range of educational attainment and diverse backgrounds. Participants complete eight behavioral assessments of executive functioning and reasoning that are unrelated to educational curricula as part of their subscription.
The research team also looked closely at a subset of nearly 70,000 subscribers who finished Lumosity’s behavioral assessments a second time after about 100 days of additional cognitive training. Testing before and after the assessments measured cognitive performance in areas such as working memory, thinking quickly, responding flexibly to task goals and both verbal and non-verbal reasoning.
“Given the size and wide age range of our sample, it was possible to test whether these age effects are influenced by education – and, importantly, to determine how the cognitive effects of educational attainment differ across the lifespan, as one’s experience with formal education recedes into the past and is supplanted by other life experiences,” the team wrote.
Bunge said that collaborating with Lumosity was a golden opportunity to analyze data from around 196,000 participants – an anonymized dataset that would have taken a lifetime to collect in a laboratory.
Did you spot it? I actually do think education can play a large role in this, but how can the researchers know what the status was of the executive functions before education? Even more: if those executive functions are stable from a certain age on, it’s even more impossible to tell.
But there is another issue, if you take a look at the abstract of the study (italic by me):
Attending school is a multifaceted experience. Students are not only exposed to new knowledge but are also immersed in a structured environment in which they need to respond flexibly in accordance with changing task goals, keep relevant information in mind, and constantly tackle novel problems. To quantify the cumulative effect of this experience, we examined retrospectively and prospectively, the relationships between educational attainment and both cognitive performance and learning. We analyzed data from 196,388 subscribers to an online cognitive training program. These subscribers, ages 15–60, had completed eight behavioral assessments of executive functioning and reasoning at least once. Controlling for multiple demographic and engagement variables, we found that higher levels of education predicted better performance across the full age range, and modulated performance in some cognitive domains more than others (e.g., reasoning vs. processing speed). Differences were moderate for Bachelor’s degree vs. High School (d = 0.51), and large between Ph.D. vs. Some High School (d = 0.80). Further, the ages of peak cognitive performance for each educational category closely followed the typical range of ages at graduation. This result is consistent with a cumulative effect of recent educational experiences, as well as a decrement in performance as completion of schooling becomes more distant. To begin to characterize the directionality of the relationship between educational attainment and cognitive performance, we conducted a prospective longitudinal analysis. For a subset of 69,202 subscribers who had completed 100 days of cognitive training, we tested whether the degree of novel learning was associated with their level of education. Higher educational attainment predicted bigger gains, but the differences were small (d = 0.04–0.37). Altogether, these results point to the long-lasting trace of an effect of prior cognitive challenges but suggest that new learning opportunities can reduce performance gaps related to one’s educational history.
Well, pointing is one way of describing it. Not a really big effect and so maybe again not really suggesting that much is another way as it’s a bit different from what they used earlier on to explain why their data and study is so interesting:
“The fact that the cognitive tests were not similar to what is learned in school is a strength of the study: It speaks to the idea that schooling doesn’t merely impart knowledge – it also provides the opportunity to sharpen core cognitive skills,”
Yeah, but this isn’t the case for the pre- en posttest of the Lumosity-bit in this study, and certainly not if you look to previous recent research rhat has shown this tool has no effect on decision-making and no effect on cognitive function beyond practice effects on the training tasks.
So what we have here is a big dataset, with no way to check if the people didn’t lie, with a big selection-element (they choose to use a brain training tool) and without any information about their functioning before they took education. But ok, we have a big dataset.