Interesting discussion paper: “a country‘s ranking in international cognitive tests might misguide its educational policy”

This is an interesting piece of work by Dutch Dinand Webbink and Sander Gerritsen (HT @JanTishauer). In their discussion paper they used data from international cognitive tests (TIMSS) and compared the ranking based on cognitive results with the estimated gains pupils can have in an educational system. A year of school time increases performance in cognitive tests with 0.2 to 0.3 standard deviations for 9- year olds and with 0.1 to 0.2 standard deviations for 13-year olds.  But they found something more, in their conclusion they elaborate:

“Remarkably, we find no association between the estimated gains in achievement and the level scores of countries. At all levels of test scores we observe countries with high achievement gains and countries with low gains in achievement. The lack of association has been found for both tests (math, science) and for both age groups. Hence, assessments of the performance of education systems based on the estimated gains in achievement often are inconsistent with performance assessments based on level scores. This inconsistency might be explained by limitations of both measures. Level scores do not distinguish between the contribution of time in school and the contribution of time out of school. The gain scores only refer to the gain in achievement in the year before the test. The inconsistency of the two measures implies that the benchmarking of education systems based on level scores might yield misleading information about the performance of education systems. Low levels of test scores, or declining trends in test scores, might not be the result of low performing education systems. High levels of test scores could mask low performing education systems. Using gain scores as an additional instrument for the assessment of the performance of education systems is likely to reduce the risk of providing misleading policy information.”

Their conclusion:  “A mere focus on test score levels is likely to yield misleading information about the performance of the education system.” If you look at the results in the paper you will be surprised which countries are now topping the list and which countries are at the bottom…

They didn’t focus on PISA in their research, but do argue that the same would be applicable.

Abstract of the discussion paper:

This study provides the first estimates of the causal effect of time in school on cognitive skills for many countries around the world, for multiple age groups and for multiple subjects. These estimates enable a comparison of the performance of education systems based on gain scores instead of level scores. We use data from international cognitive tests and exploit variation induced by school entry rules within a regression discontinuity framework. The effect of time in school on cognitive skills strongly differs between countries. Remarkably, we find no association between the level of test scores and the estimated gains in cognitive skills. As such, a country‘s ranking in international cognitive tests might misguide its educational policy. Across countries we find that a year of school time increases performance in cognitive tests with 0.2 to 0.3 standard deviations for 9-year-olds and with 0.1 to 0.2 standard deviations for 13-year-olds. Estimation of gains in cognitive skills also yields new opportunities for investigating the determinants of international differences in educational achievements.

4 thoughts on “Interesting discussion paper: “a country‘s ranking in international cognitive tests might misguide its educational policy”

  1. Dit is op X, Y of Einstein? herblogden reageerde:

    Deze post kan je lezen op mijn Engelstalige blog, maar is zo relevant dat ik hier even een linkje leg. Een van de stellingen in deze discussie paper is dat landen vergelijken op basis van cognitieve tests zoals TIMSS of PISA niet noodzakelijk de juiste manier zijn. De onderzoekers herbekijken de cijfers in functie van wat kinderen op school in een jaar bijleren. Opeens blijkt de lijst van best presterende landen helemaal herschikt.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.