Why don’t we use social media to teach? It has been a common idea amongst many different educators around the globe, but is it a good idea? This new Italian working paper by Barbetta et al that I found through a tweet – oh, irony – by Carl Hendrick, describes a randomized controlled trial that suggests it isn’t such a good idea. Do note, I didn’t write ‘shows’.
To be more precise, the researchers examined a specific method in which twitter is used to discuss literature:
The TwL method is simple. The organization TwL (central unit of the project) chooses a book, creates a specific hashtag (such as #Macbetto), and establishes a reading schedule (a chapter a week, for example). The participating schools agree to adopt the same reading schedule. Once a chapter has been read (individually or in class), students can start tweeting their preferred quotes, their comments, or their rewritings of sentences in the chapter, within the usual 140 characters limit. 5 Students use the common hashtag to tweet. They can post as many tweets as they like and can comment on other people’s tweets. Teachers, as well as members of the central unit of TwL, take part in these activities to stimulate the whole community of writers.
74 schools participated in the trial. To make sure that there wasn’t a Hawthorne effect, the researchers developed a clever approach:
Overall, 37 schools were randomly assigned to the treatment group and invited to adopt the TwL protocol from the beginning of 2016/17, while 37 schools were randomly assigned to the control group and were asked to adopt their traditional approach in reading and studying “Il fu Mattia Pascal” during first term. Both groups were requested to read the first ten chapters (out of 20) of the book in November and December 2016.
So all schools did use the approach, but not at the same time.
How did they measure the effect?
The outcome variable we use to measure the effectiveness of the TwL method is the number of correct answers to a standardized follow-up test taken by students in late December 2016 and aimed at measuring the command, knowledge, and understanding of chapters one through ten of the book “Il fu Mattia Pascal”. To get more precise estimates, we use—as the main covariate in our regression models—the individual score at a baseline test administered in October 2016 and aimed at measuring students’ command of the Italian language and their understanding of written texts. Both variables have been standardized to have mean of zero and a variance of one so that we can directly show the effect size of our estimates.
And what are the conclusions:
The comparison of students in the treatment arm of the RCT (those attending schools that adopted the TwL method) to students in the control arm of the study (students attending schools that did not adopt the protocol) not only fails to find any positive and statistically significant effect of the method on the outcome variable, but shows negative effects of the intervention on students’ achievement, some of which are significant at the usual statistical levels.
That there isn’t a difference shouldn’t be regarded as something bad, but if there are negative effects, this is more troublesome. Let’s have a look:
This is the relevant explanation:
Table 3 suggests that adopting the TwL method is more detrimental to female students than male students and to students with higher scores on the baseline test. For these subgroups, the score on the follow-up test of students in the treatment group appears to be about one-third of a standard deviation lower than the performance of students in the control group. These differences are only significant at the 10% level, so that they could represent chance findings. Nonetheless, they go together with the further evidence (not statistically significant) showing that the TwL method is more detrimental to students born in Italy and students attending lyceums than their counterparts (students born abroad and students attending technical or vocational schools, respectively).
The significance at the 10% level is probably the main reason why this is a working paper rather than a published paper in a peer reviewed journal. So it’s possible it’s harming the performance of girls and the better students. The fact that it doesn’t help the learning, while there are possible negative effects does suggest to be careful when using this didactical approach.
Abstract of the working paper:
The availability of cheap wi-fi internet connections has stimulated schools to adopt Web 2.0 platforms for teaching. Using social networks and micro-blogs, teachers aim to stimulate students’ participation in school activities and their achievement. Although anecdotal evidence shows a high level of teacher satisfaction with these platforms, only a small number of studies has produced rigorous estimates of their effects on students’ achievement. We contribute to the knowledge in this field by analyzing the impact of using microblogs as a teaching tool on the reading and comprehension skills of students. Thanks to a large-scale randomized controlled trial, we find that using Twitter to teach literature has an overall negative effect on students’ average achievement, reducing performance on a standardized test score by about 25 to 40% of a standard deviation. The negative effect is heterogeneous with respect to some students’ characteristics. More specifically, the use of this Web 2.0 application appears to have a stronger detrimental effect on students who usually perform better.