This is an interesting piece of work by Dutch Dinand Webbink and Sander Gerritsen (HT @JanTishauer). In their discussion paper they used data from international cognitive tests (TIMSS) and compared the ranking based on cognitive results with the estimated gains pupils can have in an educational system. A year of school time increases performance in cognitive tests with 0.2 to 0.3 standard deviations for 9- year olds and with 0.1 to 0.2 standard deviations for 13-year olds. But they found something more, in their conclusion they elaborate:
“Remarkably, we find no association between the estimated gains in achievement and the level scores of countries. At all levels of test scores we observe countries with high achievement gains and countries with low gains in achievement. The lack of association has been found for both tests (math, science) and for both age groups. Hence, assessments of the performance of education systems based on the estimated gains in achievement often are inconsistent with performance assessments based on level scores. This inconsistency might be explained by limitations of both measures. Level scores do not distinguish between the contribution of time in school and the contribution of time out of school. The gain scores only refer to the gain in achievement in the year before the test. The inconsistency of the two measures implies that the benchmarking of education systems based on level scores might yield misleading information about the performance of education systems. Low levels of test scores, or declining trends in test scores, might not be the result of low performing education systems. High levels of test scores could mask low performing education systems. Using gain scores as an additional instrument for the assessment of the performance of education systems is likely to reduce the risk of providing misleading policy information.”
Their conclusion: “A mere focus on test score levels is likely to yield misleading information about the performance of the education system.” If you look at the results in the paper you will be surprised which countries are now topping the list and which countries are at the bottom…
They didn’t focus on PISA in their research, but do argue that the same would be applicable.
Abstract of the discussion paper:
This study provides the first estimates of the causal effect of time in school on cognitive skills for many countries around the world, for multiple age groups and for multiple subjects. These estimates enable a comparison of the performance of education systems based on gain scores instead of level scores. We use data from international cognitive tests and exploit variation induced by school entry rules within a regression discontinuity framework. The effect of time in school on cognitive skills strongly differs between countries. Remarkably, we find no association between the level of test scores and the estimated gains in cognitive skills. As such, a country‘s ranking in international cognitive tests might misguide its educational policy. Across countries we find that a year of school time increases performance in cognitive tests with 0.2 to 0.3 standard deviations for 9-year-olds and with 0.1 to 0.2 standard deviations for 13-year-olds. Estimation of gains in cognitive skills also yields new opportunities for investigating the determinants of international differences in educational achievements.