My students have their final exams this weeks and even yesterday I received another mail with the question if a specific thing we discusses would be on the test. A new study by the University of British Columbia shows that teachers don’t have to test everything they want their students to remember – as long as the knowledge they want to convey fits together well, and the test questions are well-chosen. Still, I do think it’s important that your test is a fair representation of everything that has been discussed in class. More important: this study again shows the importance of the testing effect but with extra nuances.

From the press release (bold by me):

The finding, based on an experiment with UBC pharmacy students, builds on a proven phenomenon known as “retrieval-enhanced learning” – that the very act of recalling something reinforces it in a person’s memory. But that phenomenon also has stoked fears, demonstrated in a few recent studies, of “retrieval-induced forgetting” – that items not included on a test can be purged from students’ memories.

The UBC study, published May 26 in Advances in Health Sciences Education, suggests a way to strengthen students’ memories of new knowledge without causing “assessment fatigue.”

Researchers asked more than 150 second-year pharmacy students to study a 67-slide PowerPoint file about gastroesophageal reflux disease and peptic ulcer disease – a subject relevant to their studies but not yet covered in the curriculum. Each slide contained “bullet points” typical of slides used for university lectures.

Some students were asked to study the slides for 30 minutes in anticipation of being tested on it two weeks later. Other students were asked to study the slides for 20 minutes, with the remaining 10 minutes devoted to a 10-question quiz about the material.

Two weeks later, both groups took a larger test that included the original 10 questions, plus 30 more: 10 that were about the slides, 10 about other medical conditions not covered by the slides, and 10 more general questions about basic physiology and drug characteristics (also not covered by the slides). Most questions were multiple choice, with a couple of true/false items.

As expected, the group that took the preliminary quiz a couple of weeks before did better – 22 per cent better – on the questions that were repeated in the larger test. But that group also performed 19 per cent better on other questions based on the slides even though they were not included on the preliminary quiz.

On the other questions that were not specifically about reflux disease and peptic ulcer disease, there was no statistically significant difference between the two groups’ performance.

From those results, the researchers conclude that knowledge – whether it’s about health, law, history or anything else – doesn’t have to be tested to be remembered, if it’s closely intertwined with knowledge that is being tested.

“The accumulated evidence of retrieval-enhanced learning has led some educators to assert that comprehensive testing is the best way to get students to remember knowledge,” said lead author Kevin Eva, a Professor in the Department of Medicine and Associate Director of UBC’s Centre for Health Education Scholarship. “But that could lead to valuable class time being eaten up by assessments, creating undue stress and strain for students.”

To maximize the effectiveness of testing, Dr. Eva says, course material needs to be well-integrated – each piece should be presented as part of a larger whole, not as a collection of disconnected details. In addition, test questions should be representative of the entirety of the material that instructors want students to retain.

“Students are more likely to forget untested items in a curriculum of disassociated facts,” Dr. Eva says. “So teachers need to do their homework, and make sure that what they teach comes together into a bigger, comprehensible whole.”

Abstract of the study:

Information is generally more memorable after it is studied and tested than when it is only studied. One must be cautious to use this phenomenon strategically, however, due to uncertainty about whether testing improves memorability for only tested material, facilitates learning of related non-tested content, or inhibits memory of non-tested material. 52 second-year Pharmacy students were asked to study therapeutic aspects of gastroesophageal reflux disease and peptic ulcer disease. One group was given 30 min to study. Another was given 20 min to study and 10 min to complete a 10-item test. Two weeks later a 40-item test was delivered to both groups that contained (a) the 10 learning phase questions, (b) 10 new questions drawn from the studied material, (c) 10 new questions about therapeutics in different disease states, and (d) 10 new questions drawn from more general pharmaceutical knowledge (e.g., basic physiology and drug characteristics). Moderate to large retrieval-enhanced learning effects were observed for both questions about material that was tested (22.9% difference in scores, p < 0.05, d = 0.60) and questions about material that was studied without being tested (18.9% difference, p < 0.05, d = 0.75). Such effects were not observed for questions that were not part of the study material: therapeutic questions that addressed different disease states (1.8% difference, p > 0.7, d = 0.08) or generic pharmaceutical questions (7.4% difference, p > 0.2, d = 0.32). Being tested made it more likely that students would report reviewing the material after the initial learning session, but such reports were not associated with better test performance. The benefit of mentally retrieving information from studied material appears to facilitate the retrieval of information that was studied without being tested. Such generalization of the benefit of testing can increase the flexibility of test-based pedagogic interventions.