The fallacy of evaluating “the flipped class”

In nearly 30 years of teaching, I can’t recall another teaching innovation that has aroused such interest and rapid adoption among college faculty as the “flipped” class. Somewhat belatedly, we are now seeing studies to test whether the flipped class is effective at the college level, and how it affects student learning. For advocates and early adopters of the flip, the early report on Slate from ongoing studies at Harvey Mudd are disappointing. The flipped class appears to have no significant effect, for better or worse, on student learning.

As someone who has actually flipped his class and assessed it, I think these studies are, and must be, meaningless and futile. Assessing the “flip” will be even less meaningful than assessing the “lecture” as a mode of instruction. You can well imagine that the efficacy of a lecture will depend on the subject, content, organization, size of the class, time of day, the style of delivery, the quality of the slides (if used), ambient noise, what the instructor is wearing, and so forth. A flipped class has even more significant variables, because class time can be used in so many different ways: case studies, problem-solving, peer discussion, data analysis, writing, peer-review, internet research, and still other activities. Given this variation, how could results from one study, at one or a few colleges, with one or several classes, apply to anyone else’s flipped class?

My advice to all those considering flipping their class:  flip only as needed, because you want students to do a great learning activity that is best done as a class, with students interacting with each other and with the instructor. Such learning activities and exercises may take time to find. Assess each activity – did students learn from it? For a given learning objective, did students learn more from a particular activity or exercise, than from a lecture? Which exercise benefited which group(s) of students? It’s this kind of fine-grained assessment that will be the most useful and transferable across campuses and instructors, rather than any attempt to assess a “flip” vs a “lecture.”

Logically, what students learn will depend on what they actually do. The power of the flip is that the instructor can choose among varied learning activities to engage students with each other and with the material, receive real-time feedback on student learning, and apply timely, corrective intervention when students are most receptive. It may take time and iterations for even experienced instructors to find their groove in the flipped classroom (I speak from personal experience). Make changes in your teaching with a specific objective or purpose in mind, not because someone says that the “flip” or MOOCs or badges or the next hot thing will or will not save/transform/disrupt your classroom.

I flipped my class because my own evidence convinced me that my interactive, active-learning-laced lectures were not as effective as I wanted them to be. I flipped my class because students had difficulty applying concepts to different problems. I flipped my class because students had trouble connecting and integrating concepts from different parts of the course. I flipped my class because I wanted students to see and talk about how biology applies to real-world problems like energy, food, health and the environment. I flipped my class because I had amassed a large amount of cool case study and problem-solving material that I wanted to try in class, and the flip was the best solution I could find.


Posted in Academia, Teaching and learning biology | 2 Comments

Failure of further learning: the limits of repeated study and retrieval practice

When students encounter new information, either in a textbook or on Powerpoint slides, how much of it do they learn? How much are they able to recall over time? Do repeated study help students retain more, or learn more?

Thanks to research from cognitive psychologists such as Bjork and McDaniels, we know that studying in shorter blocks of time distributed over days and weeks works better than cramming in one long study session, that alternating or interleaving study topics is better than focusing on just one subject at a time, and above all, testing (retrieval practice) dramatically improves long-term recall. These techniques are effective at countering forgetting.

What about further learning? Students rarely, if ever, absorb 100% of new content or concepts. Depending on how well the material is presented, students will recall only a fraction of the new concepts or facts presented to them, when tested immediately after. We teachers expect that student knowledge will deepen with repeated study of the same material. Now a new study by Fritz et al. describes an effect that the authors call “failure of further learning” (FOFL). FOFL refers to the observation that little further learning occurs beyond the first recall attempt, even after repeated study of a text. Students tested after repeated restudy of the same material continue to give the same correct answers, and the same wrong answers and omissions. One explanation for FOFL is that once students form a mental model of their understanding of the new material, that becomes stubbornly fixed and difficult to alter or expand.

Can elaborative (active) study techniques overcome FOFL?

In their first experiment, Fritz et al. explored whether elaborative study techniques may help students improve upon their first recall attempt. The control group of students read two texts of approximately 1000 words each (one from Dewdney’s (1993) 200% of Nothing and another from Asimov’s (1975) Eyes on the Universe), then recalled (wrote what they could remember) and reread during the same session (week 1). In weeks 2, 3 and 4, the control group recalled and then reread the same text after recall. The elaborative study group substituted re-reading with various elaborative study methods. In week 1, they underlined and annotated the text after recall. In week 2, they diagrammed or outlined after recall. In week 3, they wrote short essay questions. After both the diagramming/outlining and the question writing, they were given the text to correct or supplement their activities.

  • Week1 – read text, recall and reread (c) vs underline/annotate (x)
  • Week3 – recall and reread (c) vs diagram/outline from memory + correct from text (x)
  • Week3 – recall and reread (c) vs write Qs from memory + correct from text (x)
  • Week4 – final recall test

Annotation, diagramming or outlining, and writing test questions are study techniques that many of us recommend to our students. Did such techniques make a difference? In a word, no.

Both groups showed significant improvement in recall of both main ideas and details from week 1 to week 4; however, the magnitude of the improvement was discouragingly small. For main ideas, the scores improved from 39% to 52% in the control group, and from 47% to 53% in the experimental group. Moreover, the elaborative study techniques made no significant difference in recall of either main ideas or details.

In other experiments, Fritz et al. show that FOFL occurs even when ideas are presented as itemized lists on Powerpoint slides (why am I not surprised). They then test the hypothesis that FOFL results from students acquiring a mental “situation model” that represents their understanding of the text. They show that FOFL does not occur when the material is presented in a way that is initially confusing or difficult to understand. The initial recall results are much lower than controls where the material is more clearly presented, and restudy sessions improve the recall results to where they become comparable to the controls. The controls do exhibit FOFL. Their last experiment shows that FOFL does not apply to short-term verbatim memorization of words and phrases, where students are tested for recall immediately after restudy. In that case, each restudy session yields significant gains.

“It is impossible for a man to learn what he thinks he already knows” – Epictetus

The authors propose that FOFL occurs because once students have constructed a “situation model,” they approach restudy sessions with the attitude that they already know what this is about, and do not actively process the information. They quote Epictetus (50–138 AD): “it is impossible for a man to learn what he thinks he already knows”. What disturbs me as a teacher is that FOFL is so stubbornly resistant to the types of active study that we think are most effective.

The big question then, is how can we overcome FOFL? Students do progress from novices to experts – over time (years), with many repetitions, practice, good coaching and learning from mistakes. Finding a way to overcome or mitigate FOFL would appear essential to make learning speedier and more efficient.

An idea for future research – can group study overcome FOFL?

I do have an idea based on an observation in the Fritz et al. paper. They stated that the lack of learning gains was unlikely to be due to any inherent difficulty in some of the concepts. Just about all the main points in that first experiment were correctly recalled by some of the students. Different students recalled different points. Can students overcome FOFL by working in groups? I would like to see a study where students first practice recall individually, then get together in groups of 3-5 students to compare notes and discuss. Repeat in subsequent weeks. Will such group work make a significant difference?


Fritz, CO, PE Morris, B Reid, R Aghdassi, CE Naven 2013. Failure of further learning: activities, structure, and meaning. Br. J. Psychol. DOI: 10.1111/bjop.12060

Posted in Teaching and learning biology | Tagged , , | 2 Comments

How to stave off the MOOCpocalypse

College faculty are not yet an endangered species, but they are pressured as never before. Full-time tenure track faculty comprise a shrinking portion of faculty at all degree-granting institutions, and the majority of faculty have part-time, adjunct or contingent faculty appointments with low salaries and no benefits (AAUP 2009 report). In the face of declining state higher education budgets, increasing demand for college education, and increasing tuition, we faculty are asked to increase our productivity, or else.

For some legislators and university administrators, the “or else” is the use of MOOCs for college credit, and the possibility of degrees based at least partly on MOOCs. The premise of MOOCs is that students can experience classes taught by “star” professors at elite universities. Class lectures can be videotaped with high production values, and delivered on-line to tens of thousands, or even millions, of students at very little additional cost per student.

IF such on-line video lectures, supplemented with computer-graded or peer-graded homeworks can substantially approach the educational experience delivered at brick-and-mortar colleges and universities, then most college and university faculty will be marginalized. Colleges and universities could vastly expand their student enrollments, reduce tuition, and shrink faculty by designing their curricula around MOOCs.

Initial results from early trials at San Jose State suggest that MOOCs may be a poor fit for less prepared, more needy students, because of the rigid, one-size-fits-all design and delivery of content by MOOCs. Most people who complete MOOCs are self-motivated, self-directed learners; most already have degrees.

Where did these self-motivated, self-directed people get their degrees? From elite colleges and flagship state universities. Indeed, the students who would seem to be able to benefit the most from MOOC instruction are the students who populate the elite colleges and universities.

MOOCs are a creation of research university faculty, and reflect one common view of teaching at research universities. For many faculty (and students), their vision of great teaching is the great lecture. The larger the audience, the better. Why waste your time and effort to prepare and deliver a great lecture to only 20 students each semester, if you can record it and deliver it to tens of thousands of students at once, over and over again with little additional effort?

But this vision is highly flawed and ultimately self-destructive. Lectures, whether live or on-line, are a passive mode of learning, and one of the least effective pedagogies, ranking below reading (see Twenty Terrible Reasons for Lecturing). Lectures do not teach essential skills such as analysis and problem-solving, or teamwork and collaboration, or professional communication. Making MOOCs the centerpiece of the curriculum also marginalizes the faculty and threatens the university’s business model. Why should students pay high tuition for an education that others can get for free, except for the privilege of getting exams proctored, and perhaps having questions answered and graded by teaching assistants?

However, criticizing the MOOC model will not be enough to stave off a pending MOOCpocalypse, or reverse the decline of full-time tenure-track faculty ranks. Faculty at research universities must clearly articulate, and visibly demonstrate, their added value in the classrooms and lecture halls. If your idea of teaching is a 50-minute lecture, how is that any better than a MOOC?

So I urge my colleagues at research universities to re-imagine what goes on in their classrooms. Use the inherent advantages of live classrooms over on-line experiences: face-to-face interactions among students, collaborative learning, building relations student-to-student, student-to-faculty. Use MOOCs to “flip” your class, so students watch video lectures to get structured content outside of class, and interact with each other and with instructors in class. Use problem-solving and formative assessments to diagnose and correct misconceptions and learning difficulties while your students are still in class. Because, as As Cathy Davidson says, “if you can be placed by a computer screen, you should!”


Posted in Academia | Leave a comment

Quantitative Reasoning – an example

I asked my intro biology students, working in groups to analyze parts of the Wolfe-Simon et al. 2011 arsenic life paper, to calculate how much bacterial growth the background level of phosphate would support:

The basal culture medium contained 3 x 10-6 (3 micromolar) phosphate; this level of phosphate was present in the no-phosphate medium used to grow GFAJ-1 cells. Marine bacteria grown under P-limited conditions contain 2 femtograms (2 x 10-15 g) phosphate per cell. How much cell growth will 3 micromolar phosphate support, in cells/mL, assuming that GFAJ-1 has the same phosphate requirements as the marine bacteria? Express your answer to 2 significant digits, in the form “3.7 x 105 cells/ml”

I saw that many were struggling with this question, so I gave the students additional information, that the molar mass of phosphate is 95, that they could round up to 100.

In the end, 51% arrived at a correct solution. Is this good, bad, satisfactory, unsatisfactory?

Only about 20% of the students are Biology majors; the rest are largely engineering majors, some biochemistry, and assorted other majors.

Posted in Teaching and learning biology | Leave a comment

A victory for college faculty in Georgia

College and university faculty should long remember February 6, 2009 as a day of infamy. On that day, Clark-Atlanta University (CAU) fired 54 faculty, 14 of them tenured, well into the spring semester, with no notice, on the pretext of an “enrollment emergency.” This week, a jury ruled that CAU acted in bad faith and in breach of contract. If CAU had won this case, academic tenure would have lost most of its traditional meaning, and faculty rights would have required life support.

My very first post on this blog told the story of my wife’s experience; she never imagined that as a tenured associate professor with a current NSF grant, in her 20th year of service to CAU, she would be fired mid-semester with no notice, asked to surrender her keys the same day, and her lab, her class, and her NSF-funded research unceremoniously shut down. She was given 4 weeks severance pay, not a full year as specified in the faculty handbook for faculty terminated for fiscal exigency, and not even the rest of her academic year contract.

She and the other tenured faculty who were terminated were stunned. CAU provided no explanation of how they were chosen, nor any chance to appeal (CAU would not abide by its own grievance procedures). The fired faculty were left struggling for explanations amid trying to salvage their careers, their families, and their health. After four years of investigations by the AAUP and the EEOC, pre-trial discovery and trial testimony, we can now recount a fuller story.

Carlton Brown, the CAU President behind the mass layoffs, was brought in by the CAU Board of Trustees, chaired by Juanita Baranco, in July 2008. By December 2008, Brown had made plans to drastically cut the faculty. He and Jeffrey Phillips, interim provost, hatched a plan to declare an “enrollment emergency.” The faculty handbook lays out procedures for firing either non-tenured or tenured faculty in the case of a “fiscal exigency” or for the closure of academic programs, in accordance with AAUP policies. Tenured faculty would be given a year’s notice, and given preference in any re-hiring. However, the faculty handbook says nothing specific about the case of an “enrollment emergency.” Brown and Phillips wanted to fire tenured faculty right away and pay them only a minimum of severance (a few weeks).

Phillips then created a “faculty productivity framework” document, and asked department chairs to complete this form for all their faculty over a weekend in early January, in secrecy. The chairs have testified that they completed these forms with no awareness of how they would be used. In my wife’s case, this form was completed by a new chair who had arrived on campus just weeks earlier. These evaluations were then supposedly used as the basis to determine who would be fired. CAU has a system of annual faculty performance evaluations, reviewed and discussed with the faculty. But they were not used. Only the faculty productivity framework, of which the faculty had no knowledge and had no chance to review for accuracy, was supposedly used. I say supposedly because at least one list of proposed faculty terminations preceded the results of the faculty productivity framework.

Once Brown and Phillips had decided on a list of people to fire, they then had to declare an enrollment emergency. They had reported to the Board of Trustees that they projected a spring 2009 enrollment of 3400 students (down sharply from 4068 in Fall 2008), and faculty layoffs would be needed to meet the anticipated budget shortfall. But the VP of Student Services, Darrin Rankin, reported to Brown that spring enrollment was exceeding projections, with 3700 students already “financially enrolled” and others still in the pipeline (Rankin affidavit, Jan 27, 2010) with over a week left in the enrolment period. To Rankin’s great surprise, Brown was displeased and said he thought everyone understood that enrollment would be capped at 3400, and ordered Rankin to immediately cease further enrollment. Rankin testified that there was no “enrollment emergency.” Nevertheless, the the mass layoffs of faculty and staff occurred on February 6, 2009, and Rankin resigned.

In a nutshell, Brown and CAU manufactured a fake “enrollment emergency” as a pretext to fire large numbers of faculty and staff without notice. Brown and CAU then used a newly created, secret evaluation process, where the primary evaluators were unaware of its intended use, and the evaluated were completely unaware of its existence, to determine who would be fired. Finally, they chose to execute the firings well after the start of the semester, disrupting academic schedules for both faculty and students, in violation of the terms of the annual appointment letter faculty sign confirming their rank, duties and pay for the next academic year.

During the trial, CAU could not mount a credible defense of their actions, and the jury found unanimously in favor of the plaintiffs, for breach of contract, negligence, and attorneys’ fees. We academic faculty can breathe a little easier for the moment, that colleges and universities cannot fire faculty without justification and without due process.

We owe a debt of thanks to the brave former CAU faculty plaintiffs for enduring four years of pain and hardship to bring this case to trial. Their names are Johnny Wilson, Lonzy Lewis, Frank Sisya, Henry Neal, and Lisa Nealy.

Posted in Academia | Tagged , , , , | 2 Comments

Will the MOOCpocalypse kill graduate science programs?

There is intense debate about what MOOCs will do to or for higher education. Recent articles by Jonathan Rees on Slate (The MOOC racket) and responses by Tim Worstall on Forbes (What MOOCS will really kill is the research university), by Jonathan Chait (College Professors are about to get really mad at President Obama) and Historiann (Historiann stumbles out of the wilderness to find the Lords of MOOC creation have successfully placed an advertorial in the Washington Post) are recent thoughts I’ve read on the topic. Jonathan Rees is one of the loudest critics of MOOCs on the grounds that they are bad for students and bad for college faculty. The fear is that, if colleges and universities award credit for MOOCs, and if large numbers of students use MOOCs instead of traditional classes to complete college degrees, the teaching faculty ranks will be drastically culled.

Tim Worstall has little sympathy for the plight of college faculty, but he points out that MOOCs could adversely affect research universities as declining tuition dollars would no longer be sufficient to subsidize faculty research. I’m not sure that he’s right about tuition subsidizing research at large research universities that receive large amounts of research grant dollars and indirect cost reimbursements. But I do agree that MOOCs may kill the research university, for an entirely different reason.

If MOOCs cause faculty numbers to diminish, I foresee massive disruption of graduate programs and the academic research enterprise, especially in the sciences. In the natural sciences, graduate students and post-doctoral fellows perform most of the research in academic labs. They constitute a highly educated, highly skilled, but low-paid labor force that makes academic research a great bargain for the nation’s research portfolio. A large fraction of science PhDs eventually become teaching faculty at one of the thousands of undergraduate colleges. Permanent positions for PhD scientists at research universities, government labs, research institutes, or industry are too few to accommodate more than a small fraction of the output of research university graduate programs. If college teaching jobs disappear, what will happen to these graduate students and postdocs?

I very much doubt that federal and industrial support for research will increase substantially. Many science PhDs already struggle to make a living as contingent faculty, or go from one temporary postdoctoral appointment to another (see this article in The Atlantic magazine about worsening career prospects for new PhDs). With no college-level teaching jobs, unemployment and underemployment for PhD-level scientists will rise to intolerable levels. How many students would then choose to undergo the rigors and privations of graduate school to earn a PhD?

If faculty ranks are the first casualty of a MOOCpocalypse, graduate programs will be inevitable follow-on casualties. A decline in our graduate programs will starve academic research labs and imperil the future of science in this country.

Posted in Academia | Leave a comment

Process of science & the scientific method

We begin our first semester introductory biology course (Biology 1510 Introduction to Biological Principles) by having students read Platt’s 1964 article on Strong Inference. Mark Hay likes T. C. Chamberlin’s paper on the Method of Multiple Working Hypotheses even better. These two essays get at the essence of good science: coming up with multiple alternative hypotheses and devising ways to test them all, as efficiently and rigorously as possible. But they are somewhat dry reading, and I’m not sure how much students actually learn from them.
I think I found an article that vividly illustrates these principles for our students. Gary Taube’s article for The Crux, a Discover magazine blog, shreds the recent, highly publicized reports about the dangers of read meat consumption and the benefits of eating chocolate. He systematically shows how these observational studies do not qualify as sound science because they do not propose and test alternative hypotheses.

Pan A, PhD, Sun Q, MD, ScD, Bernstein AM, MD, ScD, et al. Red Meat Consumption and Mortality: Results From 2 Prospective Cohort Studies. Arch Intern Med. 2012;172(7):555-563. doi:10.1001/archinternmed.2011.2287.

Posted in Teaching and learning biology | Leave a comment