Investing in early childhood education through various programs happens in many places, but thanks to James Heckman, we know that sustaining — maintaining — such investments is important, because otherwise their effects can fade. That already explains part of the title of this post, which almost sounds like the beginning of a riddle. It is exactly what researchers Shruti Jha, John List, Rebecca Royer, and Anya Samek describe in a new NBER working paper. At the heart of their paper is this question: if the effect of an educational intervention is no longer visible in test scores over time, does that also mean the effect has truly disappeared?
An experiment from ten years earlier
This story begins back in 2010 and 2011 in Chicago Heights, a relatively poor area south of Chicago. Families with children aged three to five were assigned by lottery to different groups or conditions. Some children received preschool education. Other families participated in a program for parents, the Parent Academy. In addition, there was a control group whose children did not participate in the preschool program and whose parents received no training.
The Parent Academy lasted approximately nine months and consisted of workshops in which parents learned how to support their child’s development at home. This covered both academic skills and executive functions. The underlying idea was explicitly that parents are their children’s first teachers.
Important for the rest of this post, the study also included substantial monetary incentives as part of its program. The parents would receive money for participating in the workshops, performing activities at home, and undergoing the evaluation process. The total amount could be as high as $6,900, which accounted for more than a fourth of the median household income in this population.
Moreover, there were two versions of the parent program. Some parents would get the rewards immediately. In the case of other parents, some of the money was placed in a savings account to be accessed once their children entered college.
Bye-bye differences
Initially, the researchers found effects. The parent program had primarily positive effects on executive functions, while the preschool intervention showed gains mainly in academic skills. But, as already hinted at in the introduction, those differences faded over the years. By the time COVID struck, there were no longer clear differences between the groups on standardised tests.
And then the temptation is strong to conclude that the effect has disappeared. But then the schools closed because of the pandemic…
Then came Covid
The pandemic suddenly created a situation that the researchers could not possibly have foreseen ten years earlier. Indeed, for them it was, in a sense, a cynical gift from heaven. The normal contribution of schools to children’s learning largely disappeared for an extended period. And what did the researchers find? Suddenly, differences between the three groups re-emerged.
Among children in the control group, test scores fell by approximately 0.30 standard deviations during the pandemic. For children whose parents had attended the Parent Academy years earlier, the decline was approximately 0.11 standard deviations. The difference between the two groups was statistically significant. For children who had received the preschool intervention, the decline was approximately 0.25 standard deviations, with no significant difference compared with the control group.
In other words, precisely when school temporarily disappeared as an important learning environment, the children of parents who had participated in the parent program fared relatively better.
So maybe not everything had disappeared?
The researchers’ explanation is that the original intervention may have changed something in the parents that was barely visible in their children’s test scores for years, but nevertheless remained present.
As long as schools function normally, they provide a large share of the input children need in order to learn. Differences in what parents can or do at home may therefore not necessarily be very visible. But when that school input suddenly declines sharply, what parents contribute at home can become much more important.
The authors therefore refer to latent human capital. By this, they mean knowledge, skills, or behaviours built up through an intervention whose effects only become visible when circumstances change.
Personally, I find this a fascinating idea that could turn quite a few assumptions on their head, but one that is difficult to establish without such exceptional circumstances. Perhaps a fade-out does not always mean that everything caused by an intervention has actually disappeared. Perhaps we are simply measuring at a moment when those differences are barely visible.
Here comes the nuance
It is tempting to write that a parent program from ten years ago protected children from learning loss during COVID. Unfortunately, it is not that simple. Because of the natural-experiment aspect of this study — or at least of its unexpected second part — there are several limitations.
Attrition is a problem in this group of families, too. Of the initial 676 children who qualified by age, the investigators were able to obtain the required information before and after the pandemic from only 338. This is around 50%. The investigators tried to use statistical methods like inverse probability weighting to account for attrition. However, such a statistical correction can account only for the differences that are measured.
There is another nuance worth mentioning. The positive result was driven mainly by the group for whom part of the financial reward from the original program had been deposited into a savings account for future higher education. For this group, therefore, a financial incentive remained in place even years later. For the group that had received the rewards immediately, the difference was not statistically significant. When you read something like that, your brain immediately starts coming up with all kinds of possible explanations. Well, at least mine does.
This makes the story more complicated. We cannot simply say that parents learned something ten years ago and that this knowledge suddenly resurfaced during COVID. That long-term financial incentive may also have played a role in parents’ behaviour.
Furthermore, we do not know exactly what parents did during the school closures. This was not measured. We therefore do not know whether parents who participated in the program, for example, helped their children more often, offered different learning activities, or reused techniques from the workshops they had attended ten years earlier.
Don’t get me wrong: I think the proposed explanation is more than plausible, but the mechanism itself has not been proven.
So what do we know?
What I get out of the study is that even if an effect seems to fade out, it needn’t be gone for good. There could still be something left there, which might be observable under the right conditions (no, I don’t mean cynical). Determining what those conditions are looks a lot trickier.