“To Really Learn, Quit Studying and Take a Test”, declares the alluring and misleading headline of a NY Times article by Pam Belluck, describing the findings of yet another paper on teaching and learning published by Science, on-line in Science Express (Karpicke and Blunt, 2011). However, the study described has nothing to do with testing as a method to improve learning.
The title of the Science Express paper is “Retrieval practice produces more learning than elaborative studying with concept mapping.” The study by Jeffrey Karpicke and Janelle Blunt is actually about determining what study methods work better. I found their paper compelling reading, first because they tested college students (Purdue undergraduates) learning real concepts in biology, and secondly because they rigorously tested concept-mapping, a widely accepted and promoted “active-learning” strategy, versus what they call “retrieval practice,” which turns out to be writing.
For some time now, advocates of science teaching reform have promoted “Active Learning” as an alternative to straight lectures with Powerpoint slides. I have joined this reform movement, because I hate seeing students in lecture halls sitting passively, if not nodding off, chatting with their neighbors, texting, or otherwise multitasking on their laptops. Few students these days bother to take notes. As a teacher, I want to get my students actively engaged with the material, thinking, talking, writing, and drawing science. Many studies have shown that such active learning methods produce better outcomes than lecture-only. What I dearly want to know, is whether some methods work better than others, and how I can advise students to get the most out of the time they do spend studying.
Karpicke and Blunt report results from two experiments. Experiment 1 was a 4-way design, where 4 different groups of 20 students each read a 276-word biology text on “Sea Otters.” All the groups had 5 minutes to read this text. The control “Study” group had no further study time. The “Repeated Study” group was given 3 additional 5-minute study periods to re-read the same text, with 1-minute breaks in between, so these students studied the text a total of 4 times. The “Concept Mapping” group, after they had read the text for 5 minutes, were instructed briefly in concept mapping, with an example of a concept map, then given 25 minutes with a sheet of paper to construct a concept map, while being allowed to consult both the text and the example map. The “Retrieval Practice” group, after the initial 5 minutes to read the text, were then given 10 minutes to type into a text box on the computer screen all the information they could remember from the text. They then re-read the original text for 5 minutes, then repeated the 10-minute recall typing exercise. Thus the “Concept Mapping” and “Retrieval Practice” groups had equal amounts of total learning and study time (30 minutes), but more than the “Repetitive Study” group (20 minutes), and the “Study” group (5 minutes). Assessments of the concept maps and the written recalls showed that the concept maps captured 78% of the conceptual information, and the first and second recalls captured 64% and 81%, respectively. Therefore, there was no significant difference at the end of the learning phase between the concept maps and the second recall writings.
A week later, the students took a paper-and-pencil test on this material, with 14 short answer questions to test verbatim recall (Figure 1A) and 2 questions to test whether students could make inferences based on the concepts (Figure 1B). The results showed that Retrieval Practice produced the highest test scores, by a significant margin. What is surprising, and perhaps shocking to the faculty who like to use concept mapping in their classes, is that students who did the Concept Mapping fared no better than students who spent less time with the most boring study method – repetitively re-reading the same text.
One might expect that students who did concept-mapping might fare best on inferential questions, which require students to mentally connect concepts, but Figure 1B shows the same pattern of performance as in Figure 1A. What I find remarkable is that students who did the concept mapping were the most confident about their learning – after the study period, all students were asked to predict how much they would remember, and those who did concept mapping thought they would remember the most (Figure 1C). In contrast, the students who did the Retrieval Practice were even less confident about their learning than even the control Study students, but outperformed all other groups. This clearly shows that students are not good judges of their own learning, and that student surveys are not reliable indicators of what methods are most effective at promoting student learning.
So were these results a fluke of relatively small sample size, given that each group had only 20 students? The authors used quite a different strategy for Experiment 2. Here they used a larger cohort (120 students), a head-to-head comparison of Concept Mapping vs Retrieval Practice, and a within-subjects design. Each student read 2 texts. With one of the texts, they used Retrieval Practice. With the other text, they used Concept Mapping. One additional factor thrown in was the kind of information: sequence text presented concepts that were structured like steps in a sequential process, such as “Digestive System”; and enumeration text presented concepts where order was unimportant, such as “Makeup of Human Blood”. Would Concept Mapping be superior for learning sequential information?
The actual studying procedures were similar to Experiment 1. Remember that the difference now is that each student used both study methods; Concept Mapping for one text, Retrieval Practice for another text. Again, the concept maps and the written recalls were evaluated at the end of the learning phase. This time the concept maps captured significantly more information (74%), than either the first or second recall writings (46% and 65%, respectively).
The testing phase a week later added a significant new wrinkle: half the students were asked to create a concept map (without looking at the text), and half the students were asked short-answer questions as before.
What is remarkable about the test results is their consistency. Retrieval Practice is superior, even when students are tested by Concept Mapping. The same student, performed better on the subject he or she had studied by Retrieval Practice, than on the subject he or she had studied by Concept Mapping. It didn’t matter if the subject was on sequential information or enumerative information. It didn’t matter if the test was to construct a paper-and-pencil concept map or type short written answers. Students who had studied a text by Concept Mapping did not do as well at reconstructing the concept map a week later, than students who had studied the same text by Retrieval Practice.
I can’t think of any way around the conclusion: Retrieval Practice is better than Concept Mapping. Why?
Concept Mapping is popular among active-learning enthusiasts because it engages students to make mental constructs. As performed in these experiments, it is “elaborative” study – students work with the text to add their own interpretation. Retrieval Practice seems little different from rote recall, a seemingly less interesting or lower form of active learning. But clearly something powerful happens during Retrieval Practice that promotes long-term learning.
The authors cite evidence that the act of free recall (recalling in the absence of specific prompts) forces the subject to organize a mental “retrieval structure” and then sort and recover individual concepts within the structure. The practice part allows the subject to see the gaps in the structure, and fill them in.
As someone who has always despised memorization, but is sometimes fond of writing, I wonder if the act of writing (typing into a text box) is part of this learning effect. Would Retrieval Practice work as well if the subjects orally recited their recall? What if the Concept Mapping had been performed in a way similar to Retrieval Practice – a Retrieval Concept Mapping, where students read the text, make a concept map from recall, re-read the text, and revise their concept maps, again from recall?
There’s been a few recent papers about how brief writing exercises can eliminate the gender gap in physics (Miyake et al. 2010), enhance academic performance among black students (Cohen et al. 2009), and alleviate test anxiety (Ramirez and Beilock 2011). While the student writing in these other papers are not germane to any academic subject matter, they nevertheless demonstrate that the act of writing can have powerful, long-lasting effects. Would these studies have seen the same effects if the subjects had talked about their values or anxieties rather than writing them? Does writing affect our brains in a special way?
Karpicke, JD and JR Blunt 2011. Retrieval Practice Produces More Learning than Elaborative Studying with Concept Mapping. Science DOI:10.1126/science.1199327
Cohen, GL, J Garcia, V Purdie-Vaughns, N Apfel, P Brzustoski 2009. Recursive Processes in Self-Affirmation: Intervening to Close the Minority Achievement Gap. Science 324: 400-403 http://dx.doi.org/10.1126/science.1170769
Ramirez, G and SL Beilock 2011. Writing about Testing Worries Boosts Exam Performance in the Classroom. Science 331:211-213 http://dx.doi.org/10.1126/science.1199427
Miyake, A, LE Kost-Smith, ND Finkelstein, SJ Pollock, GL Cohen, TA Ito 2010. Reducing the Gender Achievement Gap in College Science: a Classroom Study of Values Affirmation. Science 330:1234-1237 http://dx.doi.org/10.1126/science.1195996