It has something that has been shown again and again:

• Children in primary and secondary education have performed – on average! – worse during the pandemic
• Students in higher education often did better than before.

But a new NBER working paper by Altindag et al begs to differ and warns that this could well have been a mistake and the opposite is true. They examined the data of 15.000 students from an unnamed university that “… has a long-established online education program, which offers a convenient, flexible learning platform for its students.”

The researchers collected…

information on students’ midterm and final grades in both online and F2F courses. It also provides information on students’ age, gender, ethnic background, their major, zip code of their residence, and course information, including the course name, delivery method, and course level.

And although they only looked at the data of 1 university – maybe it’s different in all other institutes? – they did discover some important mechanisms that can obscure the reality, making policymakers think the students did better:

…our investigation also reveals that instructor-specific factors, such as leniency in grading or actions towards preventing violations of academic integrity, play a significant role in determining the studied relationship. Without accounting for these instructor-specific factors, the relationship is severely biased, causing one to mistakenly conclude that online instruction is better for student learning than face-to-face instruction.

The researchers do leave a bit of hope in their final conclusions:

While our paper shows that online education results in poorer student learning than F2F education in general, this finding should not be regarded as the final verdict in the debate about the merits of online versus F2F education. It is important to keep in mind that remote learning is a constantly evolving experience, driven by reliable connectivity and high-speed broadband, as well as the advancement of cloud-based technologies. Therefore, it is possible that integration of information technology in education would eventually improve student and instructor experience in ways that result in achievement gains for online over the traditional form instruction.

But if you read between the lines, that isn’t the case just yet in most institutes.

Abstract of the working paper:

The pandemic has revived the longstanding debate about the effect of online versus face-to-face instruction on student achievement. The goal of this paper is to provide new evidence on the impact of online versus face-to-face instruction on student learning outcomes, using rich, transcript-level longitudinal data from a public university. We pay particular attention to eliminating selection bias by incorporating student and instructor fixed effects into the empirical analysis as well as to separate out the impact of online versus in-person education from COVID-19-related confounding factors. Our results indicate that students in face-to-face courses perform better than their online counterparts with respect to their grades, the propensity to withdraw from the course, and the likelihood of receiving a passing grade. However, our investigation also reveals that instructor-specific factors, such as leniency in grading or actions towards preventing violations of academic integrity, play a significant role in determining the studied relationship. Without accounting for these instructor-specific factors, the relationship is severely biased, causing one to mistakenly conclude that online instruction is better for student learning than face-to-face instruction. Our analysis further documents a rise in grades associated with COVID-19-triggered changes to student assessment policies embraced by universities as well as instructors adopting a more flexible approach to grading. While these developments led to an increase in grades for all students overall, those who began Spring 2020 in face-to-face courses appear to have benefitted more generously from them. Finally, an auxiliary analysis shows that living in neighborhoods with better broadband technology is associated with a larger increase in grades among students who had to switch from in-person to online instruction during COVID-19. This finding supports the argument that unequal access to technology might have caused learning disparities to get deepened during the pandemic.