Maybe you’ve forgotten about the little turtle, but ever since Logo by Seymour Papert, programming has found its way into curricula in many countries. Sometimes that little turtle has become rather more real, with robots now also appearing in STEM classes. Little side note: one of my children once remarked that this seems to happen most often on open days.

Still, students do program robots to navigate a course, build things themselves, or use them to learn about mathematics, engineering or computer science. The idea makes sense. You do something, immediately see whether your code works, and try again. It can also be quite motivating. But does it help students learn?

We’ve been doing all of this for decades, so you might expect a reasonably clear answer by now. The good news is: there is indeed a lot of research. A new study by Sanna Forsström, Melissa Bond, and Morten Njå in Review of Education is actually a meta-review, a review of reviews. The researchers distilled the information from 49 previous research syntheses on programming and educational robotics in primary and secondary education, published between 2012 and 2023. So this concerns the most recent wave of research on this topic. And I’ll give away the conclusion for anyone who doesn’t want to read the rest: despite all that research, we still know surprisingly little for sure.

First, the good news

Sorry for throwing in a clickbait heading to keep people reading. Anyone interested in finding justifications for programming and robotics education will surely find positive evidence in the current review literature.

According to Forsström et al., at least four major advantages consistently emerge in previous reviews:

• motivation and engagement,
• skills development,
• STEM learning,
• social relationships and collaboration.

I have already mentioned the first one in my introduction. Programming and robotics can also be highly practical. The student creates a program or builds a robot, experiments with it, observes the outcomes, and modifies his/her work further based on these observations. Several studies reviewed in the past indicate, for instance, that visual programming languages may be more motivating than immediate coding in text. I believe it may be true just because of their accessibility.

The authors can also report positive results for skills. This naturally includes learning to program and computational thinking. However, other skills, such as problem-solving, logical reasoning, and creativity, could be mentioned, given the extensive literature on this topic. Even some reviews present effects that extend beyond programming skills.  For example, on mathematical skills, metacognition, or reasoning. I am personally more cautious here because, in our own work on this topic, such transfer was usually very limited or nonexistent.

Educational robotics also seems to offer possibilities for STEM education. A robot, for example, can suddenly make an abstract principle visible and tangible. You literally see what happens when your instruction is correct or incorrect. So far, the good news.

But then the problems begin

The fact that positive effects are reported doesn’t mean that we can firmly conclude that programming or using robots in the classroom improves learning. First problem: the 49 reviews are not all equally strong. The researchers assessed the methodological quality, and no fewer than 30 of the 49, or 61 per cent, fell into the medium-quality category. Seven were of high quality, and only two received the rating excellent. Ten were rated as low quality. Well, it could be worse.

However, on closer analysis, they find that only 38.8 per cent of the reviews provide full details of their inclusion and exclusion criteria. In my opinion, what is more problematic is that almost half of the reviews do not assess the quality of the studies under review. Garbage in, garbage out applies to reviews. Moreover, two-thirds reported a lack of inter-rater agreement information when coding studies.

And then there is another problem we need to discuss. The underlying studies differ enormously from one another. We’re dealing with different age groups, programming languages, types of robots, subjects, intervention durations and, above all, different ways teachers use all of this in their teaching. You are not just comparing apples with lemons, but different kinds of apples with different kinds of lemons, and then with different cuts of meat as well. Good luck combining all that.

The authors therefore state that, based on the existing reviews, they were unable to produce a complete synthesis of the effects. And many of those earlier reviews were primarily descriptive. So much for estimating the effects.

The robot is not the intervention

It is tempting to ask questions such as: Do robots work in education? Or: Does programming work? But these questions are probably way too broad and, above all, too crude. A robot sitting somewhere in a classroom does not teach anyone anything. The same goes for Scratch, Python, or any other programming environment. What matters is what students are asked or told to do with it, what prior knowledge they have, what the learning goal is, and how the teacher guides the learning process.

This is also apparent from this meta-review by Forsström and colleagues. According to the authors, the potential benefits do not arise automatically but depend on the combination of tasks, tools, and certainly the role of the teacher. Good support and scaffolding can be important here. But also consider authentic tasks, collaboration, and how a task is structured. All of these factors can have an influence.

But I deliberately wrote can. The authors found, for example, no meta-syntheses that specifically map out which pedagogical practices are successful when programming and robotics are integrated into particular school subjects. A study might therefore find positive results without telling us which pedagogical choices were responsible for them.

And there can also be disadvantages

We’re not quite finished yet. The story does not have only good news about potential outcomes. Nothing in education is free of side effects. This one is no exception. Some difficulties did crop up in the earlier analysis.

The first is rather practical: Robots and other technologies cost money, require maintenance, and should be reliable. The necessary technologies, programs, and equipment are needed. I really like those Lego robots, but I nearly fainted at their price.

Another problem concerns the cognitive load theory. Think of it: a student may have to

• understand a mathematical problem,
• learn to code,
• figure out how a particular programming environment works,
• and then also understand why the robot isn’t doing what was expected.

A tool intended to make learning more concrete can end up adding another layer of complexity.

Teachers are also confronted with a heavy load (pun intended), as they need more than technical knowledge. They need to know how to structure the task, what kind of support students need, and when that support can be gradually withdrawn. The researchers repeatedly emphasise the need for effective professional development in their meta-review.

And there is something else. We know relatively little about the long-term effects. Much of the research examines relatively short interventions. That makes it difficult to know whether an initial boost in motivation or performance will last. It may simply fade.

A lot of research is not the same as a lot of knowledge

How often do you hear researchers say that more research is needed? Guilty as charged, I say it often enough myself. But a review like this is useful because it also shows what we don’t need more of. Some things have been studied over and over again, while the more concrete questions remain unanswered.