Failure of further learning: the limits of repeated study and retrieval practice

When students encounter new information, either in a textbook or on Powerpoint slides, how much of it do they learn? How much are they able to recall over time? Do repeated study help students retain more, or learn more?

Thanks to research from cognitive psychologists such as Bjork and McDaniels, we know that studying in shorter blocks of time distributed over days and weeks works better than cramming in one long study session, that alternating or interleaving study topics is better than focusing on just one subject at a time, and above all, testing (retrieval practice) dramatically improves long-term recall. These techniques are effective at countering forgetting.

What about further learning? Students rarely, if ever, absorb 100% of new content or concepts. Depending on how well the material is presented, students will recall only a fraction of the new concepts or facts presented to them, when tested immediately after. We teachers expect that student knowledge will deepen with repeated study of the same material. Now a new study by Fritz et al. describes an effect that the authors call “failure of further learning” (FOFL). FOFL refers to the observation that little further learning occurs beyond the first recall attempt, even after repeated study of a text. Students tested after repeated restudy of the same material continue to give the same correct answers, and the same wrong answers and omissions. One explanation for FOFL is that once students form a mental model of their understanding of the new material, that becomes stubbornly fixed and difficult to alter or expand.

Can elaborative (active) study techniques overcome FOFL?

In their first experiment, Fritz et al. explored whether elaborative study techniques may help students improve upon their first recall attempt. The control group of students read two texts of approximately 1000 words each (one from Dewdney’s (1993) 200% of Nothing and another from Asimov’s (1975) Eyes on the Universe), then recalled (wrote what they could remember) and reread during the same session (week 1). In weeks 2, 3 and 4, the control group recalled and then reread the same text after recall. The elaborative study group substituted re-reading with various elaborative study methods. In week 1, they underlined and annotated the text after recall. In week 2, they diagrammed or outlined after recall. In week 3, they wrote short essay questions. After both the diagramming/outlining and the question writing, they were given the text to correct or supplement their activities.

  • Week1 – read text, recall and reread (c) vs underline/annotate (x)
  • Week3 – recall and reread (c) vs diagram/outline from memory + correct from text (x)
  • Week3 – recall and reread (c) vs write Qs from memory + correct from text (x)
  • Week4 – final recall test

Annotation, diagramming or outlining, and writing test questions are study techniques that many of us recommend to our students. Did such techniques make a difference? In a word, no.

Both groups showed significant improvement in recall of both main ideas and details from week 1 to week 4; however, the magnitude of the improvement was discouragingly small. For main ideas, the scores improved from 39% to 52% in the control group, and from 47% to 53% in the experimental group. Moreover, the elaborative study techniques made no significant difference in recall of either main ideas or details.

In other experiments, Fritz et al. show that FOFL occurs even when ideas are presented as itemized lists on Powerpoint slides (why am I not surprised). They then test the hypothesis that FOFL results from students acquiring a mental “situation model” that represents their understanding of the text. They show that FOFL does not occur when the material is presented in a way that is initially confusing or difficult to understand. The initial recall results are much lower than controls where the material is more clearly presented, and restudy sessions improve the recall results to where they become comparable to the controls. The controls do exhibit FOFL. Their last experiment shows that FOFL does not apply to short-term verbatim memorization of words and phrases, where students are tested for recall immediately after restudy. In that case, each restudy session yields significant gains.

“It is impossible for a man to learn what he thinks he already knows” – Epictetus

The authors propose that FOFL occurs because once students have constructed a “situation model,” they approach restudy sessions with the attitude that they already know what this is about, and do not actively process the information. They quote Epictetus (50–138 AD): “it is impossible for a man to learn what he thinks he already knows”. What disturbs me as a teacher is that FOFL is so stubbornly resistant to the types of active study that we think are most effective.

The big question then, is how can we overcome FOFL? Students do progress from novices to experts – over time (years), with many repetitions, practice, good coaching and learning from mistakes. Finding a way to overcome or mitigate FOFL would appear essential to make learning speedier and more efficient.

An idea for future research – can group study overcome FOFL?

I do have an idea based on an observation in the Fritz et al. paper. They stated that the lack of learning gains was unlikely to be due to any inherent difficulty in some of the concepts. Just about all the main points in that first experiment were correctly recalled by some of the students. Different students recalled different points. Can students overcome FOFL by working in groups? I would like to see a study where students first practice recall individually, then get together in groups of 3-5 students to compare notes and discuss. Repeat in subsequent weeks. Will such group work make a significant difference?


Fritz, CO, PE Morris, B Reid, R Aghdassi, CE Naven 2013. Failure of further learning: activities, structure, and meaning. Br. J. Psychol. DOI: 10.1111/bjop.12060

Posted in Teaching and learning biology | Tagged , , | 2 Comments

How to stave off the MOOCpocalypse

College faculty are not yet an endangered species, but they are pressured as never before. Full-time tenure track faculty comprise a shrinking portion of faculty at all degree-granting institutions, and the majority of faculty have part-time, adjunct or contingent faculty appointments with low salaries and no benefits (AAUP 2009 report). In the face of declining state higher education budgets, increasing demand for college education, and increasing tuition, we faculty are asked to increase our productivity, or else.

For some legislators and university administrators, the “or else” is the use of MOOCs for college credit, and the possibility of degrees based at least partly on MOOCs. The premise of MOOCs is that students can experience classes taught by “star” professors at elite universities. Class lectures can be videotaped with high production values, and delivered on-line to tens of thousands, or even millions, of students at very little additional cost per student.

IF such on-line video lectures, supplemented with computer-graded or peer-graded homeworks can substantially approach the educational experience delivered at brick-and-mortar colleges and universities, then most college and university faculty will be marginalized. Colleges and universities could vastly expand their student enrollments, reduce tuition, and shrink faculty by designing their curricula around MOOCs.

Initial results from early trials at San Jose State suggest that MOOCs may be a poor fit for less prepared, more needy students, because of the rigid, one-size-fits-all design and delivery of content by MOOCs. Most people who complete MOOCs are self-motivated, self-directed learners; most already have degrees.

Where did these self-motivated, self-directed people get their degrees? From elite colleges and flagship state universities. Indeed, the students who would seem to be able to benefit the most from MOOC instruction are the students who populate the elite colleges and universities.

MOOCs are a creation of research university faculty, and reflect one common view of teaching at research universities. For many faculty (and students), their vision of great teaching is the great lecture. The larger the audience, the better. Why waste your time and effort to prepare and deliver a great lecture to only 20 students each semester, if you can record it and deliver it to tens of thousands of students at once, over and over again with little additional effort?

But this vision is highly flawed and ultimately self-destructive. Lectures, whether live or on-line, are a passive mode of learning, and one of the least effective pedagogies, ranking below reading (see Twenty Terrible Reasons for Lecturing). Lectures do not teach essential skills such as analysis and problem-solving, or teamwork and collaboration, or professional communication. Making MOOCs the centerpiece of the curriculum also marginalizes the faculty and threatens the university’s business model. Why should students pay high tuition for an education that others can get for free, except for the privilege of getting exams proctored, and perhaps having questions answered and graded by teaching assistants?

However, criticizing the MOOC model will not be enough to stave off a pending MOOCpocalypse, or reverse the decline of full-time tenure-track faculty ranks. Faculty at research universities must clearly articulate, and visibly demonstrate, their added value in the classrooms and lecture halls. If your idea of teaching is a 50-minute lecture, how is that any better than a MOOC?

So I urge my colleagues at research universities to re-imagine what goes on in their classrooms. Use the inherent advantages of live classrooms over on-line experiences: face-to-face interactions among students, collaborative learning, building relations student-to-student, student-to-faculty. Use MOOCs to “flip” your class, so students watch video lectures to get structured content outside of class, and interact with each other and with instructors in class. Use problem-solving and formative assessments to diagnose and correct misconceptions and learning difficulties while your students are still in class. Because, as As Cathy Davidson says, “if you can be placed by a computer screen, you should!”


Posted in Academia | Leave a comment

Quantitative Reasoning – an example

I asked my intro biology students, working in groups to analyze parts of the Wolfe-Simon et al. 2011 arsenic life paper, to calculate how much bacterial growth the background level of phosphate would support:

The basal culture medium contained 3 x 10-6 (3 micromolar) phosphate; this level of phosphate was present in the no-phosphate medium used to grow GFAJ-1 cells. Marine bacteria grown under P-limited conditions contain 2 femtograms (2 x 10-15 g) phosphate per cell. How much cell growth will 3 micromolar phosphate support, in cells/mL, assuming that GFAJ-1 has the same phosphate requirements as the marine bacteria? Express your answer to 2 significant digits, in the form “3.7 x 105 cells/ml”

I saw that many were struggling with this question, so I gave the students additional information, that the molar mass of phosphate is 95, that they could round up to 100.

In the end, 51% arrived at a correct solution. Is this good, bad, satisfactory, unsatisfactory?

Only about 20% of the students are Biology majors; the rest are largely engineering majors, some biochemistry, and assorted other majors.

Posted in Teaching and learning biology | Leave a comment

A victory for college faculty in Georgia

College and university faculty should long remember February 6, 2009 as a day of infamy. On that day, Clark-Atlanta University (CAU) fired 54 faculty, 14 of them tenured, well into the spring semester, with no notice, on the pretext of an “enrollment emergency.” This week, a jury ruled that CAU acted in bad faith and in breach of contract. If CAU had won this case, academic tenure would have lost most of its traditional meaning, and faculty rights would have required life support.

My very first post on this blog told the story of my wife’s experience; she never imagined that as a tenured associate professor with a current NSF grant, in her 20th year of service to CAU, she would be fired mid-semester with no notice, asked to surrender her keys the same day, and her lab, her class, and her NSF-funded research unceremoniously shut down. She was given 4 weeks severance pay, not a full year as specified in the faculty handbook for faculty terminated for fiscal exigency, and not even the rest of her academic year contract.

She and the other tenured faculty who were terminated were stunned. CAU provided no explanation of how they were chosen, nor any chance to appeal (CAU would not abide by its own grievance procedures). The fired faculty were left struggling for explanations amid trying to salvage their careers, their families, and their health. After four years of investigations by the AAUP and the EEOC, pre-trial discovery and trial testimony, we can now recount a fuller story.

Carlton Brown, the CAU President behind the mass layoffs, was brought in by the CAU Board of Trustees, chaired by Juanita Baranco, in July 2008. By December 2008, Brown had made plans to drastically cut the faculty. He and Jeffrey Phillips, interim provost, hatched a plan to declare an “enrollment emergency.” The faculty handbook lays out procedures for firing either non-tenured or tenured faculty in the case of a “fiscal exigency” or for the closure of academic programs, in accordance with AAUP policies. Tenured faculty would be given a year’s notice, and given preference in any re-hiring. However, the faculty handbook says nothing specific about the case of an “enrollment emergency.” Brown and Phillips wanted to fire tenured faculty right away and pay them only a minimum of severance (a few weeks).

Phillips then created a “faculty productivity framework” document, and asked department chairs to complete this form for all their faculty over a weekend in early January, in secrecy. The chairs have testified that they completed these forms with no awareness of how they would be used. In my wife’s case, this form was completed by a new chair who had arrived on campus just weeks earlier. These evaluations were then supposedly used as the basis to determine who would be fired. CAU has a system of annual faculty performance evaluations, reviewed and discussed with the faculty. But they were not used. Only the faculty productivity framework, of which the faculty had no knowledge and had no chance to review for accuracy, was supposedly used. I say supposedly because at least one list of proposed faculty terminations preceded the results of the faculty productivity framework.

Once Brown and Phillips had decided on a list of people to fire, they then had to declare an enrollment emergency. They had reported to the Board of Trustees that they projected a spring 2009 enrollment of 3400 students (down sharply from 4068 in Fall 2008), and faculty layoffs would be needed to meet the anticipated budget shortfall. But the VP of Student Services, Darrin Rankin, reported to Brown that spring enrollment was exceeding projections, with 3700 students already “financially enrolled” and others still in the pipeline (Rankin affidavit, Jan 27, 2010) with over a week left in the enrolment period. To Rankin’s great surprise, Brown was displeased and said he thought everyone understood that enrollment would be capped at 3400, and ordered Rankin to immediately cease further enrollment. Rankin testified that there was no “enrollment emergency.” Nevertheless, the the mass layoffs of faculty and staff occurred on February 6, 2009, and Rankin resigned.

In a nutshell, Brown and CAU manufactured a fake “enrollment emergency” as a pretext to fire large numbers of faculty and staff without notice. Brown and CAU then used a newly created, secret evaluation process, where the primary evaluators were unaware of its intended use, and the evaluated were completely unaware of its existence, to determine who would be fired. Finally, they chose to execute the firings well after the start of the semester, disrupting academic schedules for both faculty and students, in violation of the terms of the annual appointment letter faculty sign confirming their rank, duties and pay for the next academic year.

During the trial, CAU could not mount a credible defense of their actions, and the jury found unanimously in favor of the plaintiffs, for breach of contract, negligence, and attorneys’ fees. We academic faculty can breathe a little easier for the moment, that colleges and universities cannot fire faculty without justification and without due process.

We owe a debt of thanks to the brave former CAU faculty plaintiffs for enduring four years of pain and hardship to bring this case to trial. Their names are Johnny Wilson, Lonzy Lewis, Frank Sisya, Henry Neal, and Lisa Nealy.

Posted in Academia | Tagged , , , , | 2 Comments

Will the MOOCpocalypse kill graduate science programs?

There is intense debate about what MOOCs will do to or for higher education. Recent articles by Jonathan Rees on Slate (The MOOC racket) and responses by Tim Worstall on Forbes (What MOOCS will really kill is the research university), by Jonathan Chait (College Professors are about to get really mad at President Obama) and Historiann (Historiann stumbles out of the wilderness to find the Lords of MOOC creation have successfully placed an advertorial in the Washington Post) are recent thoughts I’ve read on the topic. Jonathan Rees is one of the loudest critics of MOOCs on the grounds that they are bad for students and bad for college faculty. The fear is that, if colleges and universities award credit for MOOCs, and if large numbers of students use MOOCs instead of traditional classes to complete college degrees, the teaching faculty ranks will be drastically culled.

Tim Worstall has little sympathy for the plight of college faculty, but he points out that MOOCs could adversely affect research universities as declining tuition dollars would no longer be sufficient to subsidize faculty research. I’m not sure that he’s right about tuition subsidizing research at large research universities that receive large amounts of research grant dollars and indirect cost reimbursements. But I do agree that MOOCs may kill the research university, for an entirely different reason.

If MOOCs cause faculty numbers to diminish, I foresee massive disruption of graduate programs and the academic research enterprise, especially in the sciences. In the natural sciences, graduate students and post-doctoral fellows perform most of the research in academic labs. They constitute a highly educated, highly skilled, but low-paid labor force that makes academic research a great bargain for the nation’s research portfolio. A large fraction of science PhDs eventually become teaching faculty at one of the thousands of undergraduate colleges. Permanent positions for PhD scientists at research universities, government labs, research institutes, or industry are too few to accommodate more than a small fraction of the output of research university graduate programs. If college teaching jobs disappear, what will happen to these graduate students and postdocs?

I very much doubt that federal and industrial support for research will increase substantially. Many science PhDs already struggle to make a living as contingent faculty, or go from one temporary postdoctoral appointment to another (see this article in The Atlantic magazine about worsening career prospects for new PhDs). With no college-level teaching jobs, unemployment and underemployment for PhD-level scientists will rise to intolerable levels. How many students would then choose to undergo the rigors and privations of graduate school to earn a PhD?

If faculty ranks are the first casualty of a MOOCpocalypse, graduate programs will be inevitable follow-on casualties. A decline in our graduate programs will starve academic research labs and imperil the future of science in this country.

Posted in Academia | Leave a comment

Process of science & the scientific method

We begin our first semester introductory biology course (Biology 1510 Introduction to Biological Principles) by having students read Platt’s 1964 article on Strong Inference. Mark Hay likes T. C. Chamberlin’s paper on the Method of Multiple Working Hypotheses even better. These two essays get at the essence of good science: coming up with multiple alternative hypotheses and devising ways to test them all, as efficiently and rigorously as possible. But they are somewhat dry reading, and I’m not sure how much students actually learn from them.
I think I found an article that vividly illustrates these principles for our students. Gary Taube’s article for The Crux, a Discover magazine blog, shreds the recent, highly publicized reports about the dangers of read meat consumption and the benefits of eating chocolate. He systematically shows how these observational studies do not qualify as sound science because they do not propose and test alternative hypotheses.

Pan A, PhD, Sun Q, MD, ScD, Bernstein AM, MD, ScD, et al. Red Meat Consumption and Mortality: Results From 2 Prospective Cohort Studies. Arch Intern Med. 2012;172(7):555-563. doi:10.1001/archinternmed.2011.2287.

Posted in Teaching and learning biology | Leave a comment

Assessing the flipped classroom

At Pearson’s 10th Biology Leadership Conference, I’ve been amazed and impressed by how many instructors have either flipped their intro biology class, or are seriously contemplating it. There was so much interest that over 30 instructors voluntarily gave up one hour of the two measly hours of free time for an impromptu discussion of the flipped classroom.

One recurring question is how effective is the flipped classroom? Is it better than the active learning strategies that most here at the BLC already employ? Who benefits? Does one assess student learning in a flipped class the same way as in a traditional mostly-lecture class?

I’ve been very interested in assessment of my flipped class. Given high levels of student resistance and decline in my teaching evaluations by the students, the temptation is to give up and revert to my previous active learning strategies, if I find no evidence for student learning gains. For comparison with previous years, I had to stick with multiple-choice exam questions to measure student learning.

First, I gleaned isomorphic questions from the Module 3 midterm from Fall 2009 and Fall 2011 (the first semester I taught this module with the flipped model). With only 7 such questions, I found no significant differences in a T-test (mean F09 = 73.3; mean F11 = 74.7; p = 0.55). Not encouraging. But I noticed that these were mostly lower-level questions testing recall of basic concepts. The reason I flipped the class was that students were not performing as well as I like at higher Bloom’s taxonomy level questions – the application and analysis.

I then compared the Fall 2010 and Fall 2012 Midterm 3 exams. I chose the Fall 2010 exam because it had a similar proportion (about 2/3) of application/analysis questions as the Fall 2012 exam. Moreover, the Fall 2012 exam was crafted around several scenarios, with groups of questions around a single topic. The Fall 2010 exam also had clusters of questions, although not to the same extent as the Fall 2012 exam. In the interest of sharing and open education, the two midterm exams are attached to this post as pdf files.

I classified each question on the two exams as to whether they tested primarily recall (Bloom’s levels 1/2) or application/analysis (Bloom’s levels 3/4). Then I analyzed student performance on these exam questions and charted the mean performance for each type of question.


This figure shows that in the two midterms, students had much greater difficulty with the application/analysis questions than on the recall questions, regardless of year. On recall questions, student score averages in Fall 2010 and Fall 2012 were nearly identical. But students who experienced the flipped classroom in 2012 performed significantly better on application/analysis questions than students in Fall 2010 (p < 0.05, one-tailed T-test).

This evidence, limited as it is, will help me make a case to my future students for the flipped model. Moreover, I’m still in the early stages of figuring out the flipped model. I’m getting some good ideas to refine lecture videos (although I’m not convinced that lecture videos are that important), and have some ideas to improve what we do in class, with better learning activities and classroom response systems like Learning Catalytics. And I’m more excited than ever about the untextbook idea to encourage students to personalize and own their own electronic class notes.

Earlier posts about my flipped intro bio class

Midterm 3 exams: click links below to download pdfs



Posted in Teaching and learning biology | Tagged , | 10 Comments