I was in Canada this weekend at an education conference, and I noticed how many people referred to the podcast Sold a Story. I’ve already seen it pop up a few times on social media here, too, especially the episode in which Emily Hanford discusses Project Follow Through. Some people will probably be familiar with this massive American education project from the 1960s and 70s by now. It compared various approaches to educating young children from disadvantaged families. Direct Instruction (DI), the program developed by Siegfried Engelmann and colleagues, came out remarkably well. Btw, this is a specific form of direct/explicit instruction. Robert Kaplinsky published an interesting fact-check of this episode. Let me just… uh, fact-check the fact-check.

Don’t expect a major debunking in this post, but rather some extra nuance. Kaplinsky went back to the original reports, and his fact-check is absolutely worth reading. That is the main reason for this post. After all, Project Follow Through was not the beautiful, clean experiment it is sometimes made out to be. Something I am sometimes guilty of myself when I have to explain it too briefly.

It is and remains a ridiculously large experiment, or rather, a quasi-experiment. However, there are quite a few limitations:

• Schools were not randomly assigned to programs.
• Implementation varied significantly between schools.
• It’s hard to tell how faithfully schools implemented the programs.
• Debate surrounded both the chosen outcome measures and the statistical analyses.

Kaplinsky is therefore right to warn against an overly simplistic narrative: America organised the largest educational experiment ever, Direct Instruction won, end of discussion.

But wait, because the fact-check also deserves some fact-checking.

First, Kaplinsky describes very few major factual errors in Hanford’s account of Project Follow Through. On several points, the original sources simply confirm what she says. Also, criticism of the original evaluation didn’t erase Direct Instruction’s strong results.

Second, and more importantly, Project Follow Through is not the only research we have on Direct Instruction by now. The project is almost half a century behind us. Since then, DI has been studied much more extensively, including in reviews and meta-analyses (see, among others, here). Therefore, demonstrating that Project Follow Through had significant methodological limitations does not mean the evidence for Direct Instruction disappears.

Perhaps, then, the better conclusion is that Project Follow Through found strikingly positive results for Direct Instruction, which also fit within a much broader body of later research showing positive effects for DI.

Exactly: nuance upon nuance.