When the latest PISA-results were published, I wrote a blog post that warned about something that could have influenced the measurements: the switch from paper based assessment to computer based assessment. Last month Andreas Schleicher suddenly admitted that this could be the case.
Now I found this new article via Pasi Sahlberg. It’s an analysis by Hikaru Komatsu & Jeremy Rappleye to be published in the summer edition of Compare: A Journal of Comparative and International Education. In the article both authors focused on the influence this turn to technology could have had on participating countries in East Asia. For this they looked at different countries.
Test score changes during 2012- 2015 were distributed widely for countries where the time spent using the Internet was more than 20 minutes per day (Figure 1a). Test score changes were positive for some countries, but negative for other countries. The mean (plus minus standard deviation) of the test score changes for these countries was – 0.667 (plus minus 12.7) However, test score changes for countries where the time spent using the Internet was no more than 20 minutes were negative in most cases (the area surrounded by a circle in Figure 1a). Among the 11 countries, Finland and Germany were the exceptions but their score changes were not strongly positive (i.e., 2 points and 1 point, respectively). The mean (plus minus standard deviation) of the test score changes for these countries was –8.00 (plus minus 8.73). Even when we replace time spent using the Internet with alternative data – the number of computers utilized for educational purposes per student in the school we obtained qualitatively the same results.
Or even shorter:
Our results largely confirm the concerns of policymakers and mainstream media outlets in East Asia: introducing computer-based testing is a probable reason for lower PISA 2015 reading scores across the region. As a purely technical issue these results are significant for many reasons. But the most important among these is that the shift to CBT in PISA 2015 may have invalidated the OECD’s attempts to provide comparable longitudinal data to the world.