MOOC, the disaster movie?

It’s not a fun week for technology and education. Yesterday there was the often mis-quoted OECD-report and than there is this headline: “Students learn 6 times more with CMU’s Simon Initiative approach than with MOOCs”.

The study hasn’t been published in a journal, but in the Proceedings of the Second (2015) ACM Conference on Learning @ Scale, and when you examine the paper rather than the headlines of a press release, again the story is slightly different.

In fact the study compares a rather passive version of online learning, bringing back MOOC’s to watching lectures online to a more interactive approach of online learning:

CMU’s Simon Initiative, named for the late Nobel and Turning laureate Herbert Simon, aims to measurably improve student learning outcomes by harnessing a learning engineering ecosystem that has developed over several decades at CMU. The Simon Initiative approach uses CMU’s Open Learning Initiative (OLI) courses, which are built to mimic intelligent tutors in order to provide adaptive feedback and hints during learning by doing.

“Learning by doing gives students deliberative practice opportunities to address a course’s objectives,” said Ken Koedinger, professor of human-computer interaction and psychology and co-coordinator of the Simon Initiative. “With OLI, students get immediate feedback. If they do not master a concept, they have to go back to re-watch or re-read and then demonstrate they have learned before they are able to move on.”

So, if you know this and Koedinger and his team set out to understand the difference between MOOCs and OLI courses, specifically whether OLI features help students learn more than MOOC lecture videos, than even without reading the paper you would have guessed the answer?

Well, correct:

They compared two uses of an Introduction to Psychology as a Science class: 18,645 students took it as a MOOC only, while 9,075 enrolled in it as a combined MOOC and OLI course. Eleven weekly quizzes and a final exam were given to all students.

First, the researchers compared how each group’s students performed on the final exam. MOOC-only students had an average score of 57 percent, and the MOOC and OLI students averaged 66 percent. This significant difference remains after adjusting for other contributors to student success including their prior educational background and their incoming psychology knowledge.

“Do students learn more with OLI? The answer is a clear and resounding ‘yes,'” Koedinger said.

Then, the team investigated how different patterns of student use corresponded with different student learning outcomes. They found that while more watching, reading and doing all predict better learning outcomes, the amount of learning associated with each activity done was six times greater than for each video watched or page read.

Because MOOCs have a history of retention problems, the researchers also compared course dropout and completion rates. They found that MOOC and OLI students were 30% more likely to finish the course and take the final exam than those in the MOOC-only class. Also, participation in the weekly quizzes — meaning students stayed in the courses longer — was always higher for the MOOC and OLI students.

We’ve seen before that MOOC’s are often rather window dressing than well designed courses,  so this conclusion by Koedinger can hardly be a surprise:

“Most of MOOCs’ attention has been on scaling teaching — making lectures available to more people,” said Norman Bier, director of OLI and executive director of the Simon Initiative. “This study shows that students can be better served if educators and course creators focus on what we can scale — learning. More attention needs to be placed on designing, developing and improving the learning experience in online courses — with a focus on learning by doing that is well-aligned with outcomes and assessments.”

So this paper, besides being a story about how “our approach is better”, is in fact telling us that if you think putting video’s – and let’s face it often really bad video’s – online in an online open course without interaction or other kinds of didactical approaches, you’re in for a fail. It doesn’t tell us that it’s impossible to design a distance learning platform that actually can work for some students. In fact it says that an better designed MOOC can be six times better. Badly designed MOOC’s can be a disaster. No Shit, Sherlock.

Still, there are 2 other issues with this study:

  • it’s all about correlation, although they added mechanisms to support the causal claims
  • a control group with students following the same course in a non-distance approach would have been very interesting.

Abstract of the study:

The printing press long ago and the computer today have made widespread access to information possible. Learning theorists have suggested, however, that mere information is a poor way to learn. Instead, more effective learning comes through doing. While the most popularized element of today’s MOOCs are the video lectures, many MOOCs also include interactive activities that can afford learning by doing. This paper explores the learning benefits of the use of informational assets (e.g., videos and text) in MOOCs, versus the learning by doing opportunities that interactive activities provide. We find that students doing more activities learn more than students watching more videos or reading more pages. We estimate the learning benefit from extra doing (1 SD increase) to be more than six times that of extra watching or reading. Our data, from a psychology MOOC, is correlational in character, however we employ causal inference mechanisms to lend support for the claim that the associations we find are causal.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.