I have put it off long enough now. Over the past two months, I have been avoiding writing posts about research in AI and Education. Until now. In this post, I want to discuss this new meta-analysis by Zhang and colleagues in Review of Education. Their research focuses on this central question: ‘Does generative AI work for learning a foreign language?’

This new study appears to have some good news for those interested in using artificial intelligence in language education. Indeed, there is much to be said about the research. However, I will mostly explain why one needs to look beyond the positive effects.

First, I want to give you the lowdown. In this meta-analysis, 48 controlled studies from 2023 to 2026 were selected, combining a sample size of 5,196 participants and 67 effect sizes. The authors used only studies on generative AI models, such as ChatGPT. They excluded older, rule-driven chatbots.

The mean effect on second language acquisition was remarkably high: Hedges’ g = 0.76. This is close to an effect size of 0.8, which is generally considered large. The researchers also examined whether it mattered how AI was used. A high impact was recorded in speaking (g = 1.48), although vocabulary (g = 0.66) and writing (g = 0.61) showed slightly lower impacts. Notably, no statistically significant impact was discovered for reading.

The way students engage with AI is also a factor, in a way I didn’t expect. The research found that self-directed interactions had more impact than those where the action was largely dictated by AI. The most significant learning impact was seen where AI was used for feedback and assessment. So much for the good news. However, if you stop reading here, you’ll miss some crucial limitations.

A big effect, but how big exactly?

First, there is a very large level of heterogeneity between the different studies used in this analysis. The level of heterogeneity (I²) was 92.64%. This means that there were significant differences in the results, making it hard to predict the mean effect that would be observed in a real-life classroom. There were also several outliers among the included studies with large effect sizes. Effect sizes of 5.70, 5.16, and 4.56 were identified. Such values are rare in education studies.

Therefore, it is only natural for the researchers to check how removing these outliers from the analysis affects their mean effect sizes. The mean then dropped from 0.76 to 0.57, which is obviously still a fairly large and positive effect. For speaking skills, the reduction was even more striking, from 1.48 to 0.65.

Second, the researchers also found indications of possible publication bias. It is possible that fewer studies show smaller or negative results, but there are other explanations for the asymmetry as well. I do think this is another reason to be cautious about the average effect size.

How good were the original studies?

We previously saw that errors can occur during the quality control of studies included in meta-analyses. In this case, the authors did indeed assess the quality of the included studies. For the 25 randomised experiments, they used the well-known Cochrane Risk of Bias instrument (RoB 2), while for the remaining studies they used ROBINS-I. That is a strong point of this meta-analysis and one reason I am blogging about this research.

What I find missing is an analysis that actually uses those quality assessments to further investigate the results. What happens, for example, when we only include the studies with a low risk of bias? Does the average effect hold up? And do the same AI applications continue to score best?

As far as I can tell, I cannot find an answer to those questions. The researchers do conduct sensitivity analyses in which, as mentioned, extreme effect sizes are removed, but an outlier is of course not necessarily the same as a methodologically weak study. That is unfortunate, especially since the researchers themselves acknowledge that the experimental and control groups were not always sufficiently comparable… And that brings us to the next point.

What are we actually comparing?

Suppose that students in an experiment are allowed to use ChatGPT to improve their texts. As a result, they receive immediate feedback, can ask questions, and revise their texts multiple times. What does the control group do? These students write the same text but receive no comparable support. My question now is: if the first group subsequently scores better, what exactly have we demonstrated with this study?

That generative AI works? Or rather that students perform better when they receive more feedback, can practice more, or receive extra support?

In their analyses, the authors distinguish different types of control groups, including traditional instruction, other technologies, and independent work without support. But this does not necessarily make the experimental conditions comparable.

It is therefore quite possible that generative AI is a useful tool for making certain forms of support more accessible. That would be interesting in itself. However, that is not the same as demonstrating that generative AI works better than other ways of providing that same support.

Have the students actually learned something?

A final consideration concerns the difference between performing and learning. I suspect that this distinction is particularly important in research into the use of generative AI. A student who writes a better text with the help of ChatGPT has not necessarily learned to write better. Perhaps the student has indeed learned something from the feedback. But perhaps AI has primarily helped to improve the text.

The same applies to speaking skills: a student can hold a conversation with the support of a chatbot, but can that student do so without that support a few weeks later? To investigate this, we need, among other things, delayed post-tests. However, of the 48 studies in this meta-analysis, only four used these.

A few other considerations

I have a few other limitations to mention. Most studies used self-developed tests. Only thirteen used standardised tests. Furthermore, 39 of the 48 studies took place in higher education and more than half came from China, Hong Kong, Macau, or Taiwan. This limits generalizability.

So does AI work or not?

You might argue that I am splitting hairs, but there is a reason why I haven’t blogged about AI in the classroom for over two months. The quality of many studies simply isn’t good enough. This meta-analysis is solid, but I fear the large effect size will once again take on a life of its own. Let me be clear: these reservations do not prove that generative AI does not help with learning a foreign language. Conversely, the meta-analysis offers several indications of the possible positive role AI can play, particularly in relation to feedback, writing, and speaking. I have used the term ‘indications’ on purpose because the evidence does not appear conclusive to me.