Please read the question

Question: What are "spaced practice", "varied practice", and "interleaved practice"? Give a definition for each. Then give an example of each from your own experience as a learner.

page 1:

page 2

page 3

page 4

page 5

page 6

page 7

page 8

page 9

page 10

page 11

page 12

page 13

page 14

page 15

page 16

page 17

page 18

page 19

page 20

page 21

page 22

2 To Learn, Retrieve Ebersold is tall, slender, and counts among his forebears the Dakota chiefs named Wapasha and the French fur traders named Rocque who populated this part of the Mississippi River Valley where the Mayo brothers would later found their famous clinic. Ebersold's formal training included four years of college, four years of medical school, and seven years of neurosurgery training-building a foundation of knowl- edge and skills that has been broadened and deepened through continuing medical education classes, consultations with his colleagues, and his practice at the Mayo Clinic and elsewhere. He carries himself with a midwestern modesty that belies a career that counts a long list of high-profile patients who have sought out his services. When President Ronald Reagan needed treatment for injuries after a fall from his horse, Ebersold par- ticipated in the surgery and postsurgical care. When Sheikh Zayed bin Sultan Al Nahyan, president of the United Arab Emirates, needed delicate spinal repair, he and what seemed like half the nation's ministry and security forces settled in Rochester while Mike Ebersold made the repair and oversaw Zayed's recovery. Following a long career at Mayo, Mike had returned to help out at the clinic in Wisconsin, feeling indebted to it for his early medical training. The hunter whose bad luck put him in the way of an errant 12-gauge slug was luckier than he likely knows that Mike was on the job that day. The bullet had entered an area of the skull beneath which there is a large venous sinus, a soft-tissue channel that drains the brain cavity. As he examined the hunter, Ebersold knew from experience that when he opened up the wound, there was a high probability he would find this vein was torn. As he described it, MIKE EBERSOLD GOT CALLED into a hospi- tal emergency room one afternoon late in 2011 to examine a Wisconsin deer hunter who'd been found lying unconscious in a cornfield. The man had blood at the back of his head, and the men who'd found and brought him in supposed he'd maybe stumbled and cracked his skull on something. Ebersold is a neurosurgeon. The injury had brain protrud- ing, and he recognized it as a gunshot wound. The hunter re- gained consciousness in the ER, but when asked how he'd hurt himself, he had no idea. Recounting the incident later, Ebersold said, "Somebody from some distance away must have fired what appeared to be a 12-gauge shotgun, which arced over God only knows what distance, hit this guy in the back of his head, fractured his skull, and lodged into the brain about an inch. It must have been pretty much spent, or it would have gone deeper." ! You say to yourself, This patient is going to need surgery. There's brain coming out of the wound. We have to clean this up and repair this as best we can, but in so doing we may get into this big vein and that could be very, very serious." So you go through the checklist. You say, "I might need a blood trans- fusion for this patient," so you set up some blood. You review the steps, A, B, C, and D. You set up the operating room, tell- ing them ahead of time what you might be encountering. All of this is sort of protocol, pretty much like a cop getting ready to pull over a car, you know what the book says, you've gone through all these steps. Then you get to the operating room, and now you're still in this mode where you have time to think through it. You say, "Gee, I don't want to just go and pull that bullet out if there might be major bleeding. What I'll try to do is I'll work around the edges and get things freed up so I'm ready for what could go wrong, and then I'll pull it out." can't just put a stitch around it, because when you tighten it, the tissue tears, and the ligature leaks. Working urgently and mechanically, he fell back on a technique he'd developed out of necessity in past surgeries involving this vein. He cut two little pieces of muscle, from where the patient's skin had been opened up in surgery, and imported them to the site and stitched the ends of the torn vein to them. These plugs of muscle served to close the vein without deflecting its natural shape or tearing its tissue. It's a solution Mike has taught himself-one he says you won't find written anywhere, but handy in the moment, to say the least. In the sixty or so sec- onds it took to do, the patient lost another two hundred cubic centimeters of blood, but once the plugs were in place, the bleeding stopped. Some people can't tolerate this sinus vein being closed off. They get increased brain pressure because the blood doesn't drain properly. But this patient was one of the fortunate who can. The hunter left the hospital a week later. He was minus some peripheral vision but otherwise re- markably unscathed from a very close brush with mortality. Reflection Is a Form of Practice It turned out that the bullet and bone were lodged in the vein, serving as plugs, another lucky turn for the hunter. If the wound hadn't corked itself in the field, he would not have lived for more than two or three minutes. When Ebersold re- moved the bullet, the fractured bone chips fell away, and the vein let loose in a torrent. Within five minutes, you've lost two or so units of blood and now you sort of transfer out of the mode where you're thinking through this, going through the options. Now it becomes reflex, mechanical. You know it's going to bleed very, very much, so you have a very short time. You're just thinking, 'I have to get a suture around this struc- ture, and I know from previous experience I have to do it in this particular way.'" The vein in question, which is about the size of an adult's small finger, was torn in several places over a distance of about an inch and a half. It needed to be tied off above and below the rupture, but it's a flat structure that he knows well: you What inferences can we draw from this story about how we learn and remember? In neurosurgery (and, arguably, in all aspects of life from the moment you leave the womb), there's an essential kind of learning that comes from reflection on personal experience. Ebersold described it this way: A lot of times something would come up in surgery that I had difficulty with, and then I'd go home that night thinking about what happened and what could I do, for example, to improve the way a suturing went. How can I take a bigger bite with my needle, or a smaller bite, or should the stitches be closer to- gether? What if I modified it this way or that way? Then the next day back, I'd try that and see if it worked better. Or even if it wasn't the next day, at least I've thought through this, and in so doing I've not only revisited things that I learned from lectures or from watching others performing surgery but also I've complemented that by adding something of my own to it that I missed during the teaching process. Reflection can involve several cognitive activities that lead to stronger learning: retrieving knowledge and earlier training from memory, connecting these to new experiences, and visu- alizing and mentally rehearsing what you might do differently next time. It was this kind of reflection that originally had led Eber- sold to try a new technique for repairing the sinus vein at the back of the head, a technique he practiced in his mind and in the operating room until it became the kind of reflexive maneu- ver you can depend on when your patient is spouting blood at two hundred cubic centimeters a minute. To make sure the new learning is available when it's needed, Ebersold points out, you memorize the list of things that you need to worry about in a given situation: steps A, B, C, and D," and you drill on them. Then there comes a time when you get into a tight situation and it's no longer a matter of thinking through the steps, it's a matter of reflexively taking the cor- rect action. Unless you keep recalling this maneuver, it will not become a reflex. Like a race car driver in a tight situation or a quarterback dodging a tackle, you've got to act out of re- flex before you've even had time to think. Recalling it over and over, practicing it over and over. That's just so important." The Testing Effect A child stringing cranberries on a thread goes to hang them on the tree, only to find they've slipped off the other end. With- out the knot, there's no making a string. Without the knot there's no necklace, there's no beaded purse, no magnificent tapestry. Retrieval ties the knot for memory. Repeated retrieval snugs it up and adds a loop to make it fast. Since as far back as 1885, psychologists have been plotting "forgetting curves" that illustrate just how fast our cranberries slip off the string. In very short order we lose something like 70 percent of what we've just heard or read. After that, forget- ting begins to slow, and the last 30 percent or so falls away more slowly, but the lesson is clear: a central challenge to im- proving the way we learn is finding a way to interrupt the pro- cess of forgetting. The power of retrieval as a learning tool is known among psychologists as the testing effect. In its most common form, testing is used to measure learning and assign grades in school, but we've long known that the act of retrieving knowledge from memory has the effect of making that knowledge easier to call up again in the future. In his essay on memory, Aristotle wrote: "exercise in repeatedly recalling a thing strengthens the memory." Francis Bacon wrote about this phenomenon, as did the psychologist William James. Today, we know from empiri- cal research that practicing retrieval makes learning stick far better than reexposure to the original material does. This is the testing effect, also known as the retrieval-practice effect.3 To be most effective, retrieval must be repeated again and again, in spaced out sessions so that the recall, rather than becoming a mindless recitation, requires some cognitive ef- fort. Repeated recall appears to help memory consolidate into a cohesive representation in the brain and to strengthen and multiply the neural routes by which the knowledge can later be retrieved. In recent decades, studies have confirmed what Mike Ebersold and every seasoned quarterback, jet pilot, and teenaged texter knows from experiencethat repeated re- trieval can so embed knowledge and skills that they become reflexive: the brain acts before the mind has time to think. Yet despite what research and personal experience tell us about the power of testing as a learning tool, teachers and stu- dents in traditional educational settings rarely use it as such, and the technique remains little understood or utilized by teach- ers or students as a learning tool in traditional educational settings. Far from it. In 2010 the New York Times reported on a scientific study that showed that students who read a passage of text and then took a test asking them to recall what they had read retained an astonishing 50 percent more of the information a week later than students who had not been tested. This would seem like good news, but here's how it was greeted in many online neurosurgeon. The frustration many people feel toward stan- dardized, "dipstick tests given for the sole purpose of mea- suring learning is understandable, but it steers us away from appreciating one of the most potent learning tools available to us. Pitting the learning of basic knowledge against the de- velopment of creative thinking is a false choice. Both need to be cultivated. The stronger one's knowledge about the subject at hand, the more nuanced one's creativity can be in address- ing a new problem. Just as knowledge amounts to little with- out the exercise of ingenuity and imagination, creativity ab- sent a sturdy foundation of knowledge builds a shaky house. comments: "Once again, another author confuses learning with recalling information." "I personally would like to avoid as many tests as possible, especially with my grade on the line. Trying to learn in a stress- ful environment is no way to help retain information." Studying the Testing Effect in the Lab The testing effect has a solid pedigree in empirical research. The first large-scale investigation was published in 1917. Children in grades 3, 5, 6, and 8 studied brief biographies from Who's Who in America. Some of them were directed to spend varying lengths of the study time looking up from the material and silently reciting to themselves what it contained. Those who did not do so simply continued to reread the ma- terial. At the end of the period, all the children were asked to write down what they could remember. The recall test was repeated three to four hours later. All the groups who had engaged in the recitation showed better retention than those who had not done so but had merely continued to review the material. The best results were from those spending about 60 percent of the study time in recitation. A second landmark study, published 1939, tested over three thousand sixth graders across Iowa. The kids studied six-hundred-word articles and then took tests at various times before a final test two months later. The experiment showed a couple of interesting results: the longer the first test was de- layed, the greater the forgetting, and second, once a student "Nobody should care whether memorization is enhanced by practice testing or not. Our children cannot do much of any- thing anymore.+ Forget memorization, many commenters argued; education should be about high-order skills. Hmmm. If memorization is irrelevant to complex problem solving, don't tell your had taken a test, the forgetting nearly stopped, and the stu- dent's score on subsequent tests dropped very little.5 Around 1940, interest turned to the study of forgetting, and investigating the potential of testing as a form of retrieval practice and as a learning tool fell out of favor. So did the use of testing as a research tool: since testing interrupts forgetting, you can't use it to measure forgetting because that "contami- nates the subject. Interest in the testing effect resurfaced in 1967 with the publication of a study showing that research subjects who were presented with lists of thirty-six words learned as much from repeated testing after initial exposure to the words as they did from repeated studying. These resultsthat testing led to as much learning as studying didchallenged the re- ceived wisdom, turned researchers' attention back to the po- tential of testing as a learning tool, and stimulated a boomlet in testing research. In 1978, researchers found that massed studying (cram- ming) leads to higher scores on an immediate test but results in faster forgetting compared to practicing retrieval. In a sec- ond test two days after an initial test, the crammers had for- gotten 50 percent of what they had been able to recall on the initial test, while those who had spent the same period prac- ticing retrieval instead of studying had forgotten only 13 per- cent of the information recalled initially. A subsequent study was aimed at understanding what ef- fect taking multiple tests would have on subjects' long-term retention. Students heard a story that named sixty concrete objects. Those students who were tested immediately after exposure recalled 53 percent of the objects on this initial test but only 39 percent a week later. On the other hand, a group of students who learned the same material but were not tested at all until a week later recalled 28 percent. Thus, taking a single test boosted performance by 11 percent after a week. But what effect would three immediate tests have relative to one? Another group of students were tested three times after initial exposure and a week later they were able to recall 53 percent of the objectsthe same as on the initial test for the group receiving one test. In effect, the group that received three tests had been "immunized" against forgetting, com- pared to the one-test group, and the one-test group remem- bered more than those who had received no test immediately following exposure. Thus, and in agreement with later research, multiple sessions of retrieval practice are generally better than one, especially if the test sessions are spaced out. In another study, researchers showed that simply asking a subject to fill in a word's missing letters resulted in better memory of the word. Consider a list of word pairs. For a pair like foot-shoe, those who studied the pair intact had lower sub- sequent recall than those who studied the pair from a clue as obvious as foot-s_ _e. This experiment was a demonstration of what researchers call the "generation effect." The modest effort required to generate the cued answer while studying the pairs strengthened memory of the target word tested later (shoe). Interestingly, this study found that the ability to recall the word pair on later tests was greater if the practice retrieval was de- layed by twenty intervening word pairs than when it came im- mediately after first studying the pair. Why would that be? One argument suggested that the greater effort required by the delayed recall solidified the memory better. Researchers began to ask whether the schedule of testing mattered. The answer is yes. When retrieval practice is spaced, allow- ing some forgetting to occur between tests, it leads to stronger long-term retention than when it is massed. Researchers began looking for opportunities to take their inquiries out of the lab and into the classroom, using the kinds of materials students are required to learn in school. Studying the Testing Effect "In the Wild" In 2005, we and our colleagues approached Roger Cham- berlain, the principal of a middle school in nearby Columbia, Illinois, with a proposition. The positive effects of retrieval practice had been demonstrated many times in controlled lab- oratory settings but rarely in a regular classroom setting. Would the principal, teachers, kids, and parents of Colum- bia Middle School be willing subjects in a study to see how the testing effect would work in the wild"? Chamberlain had concerns. If this was just about memori- zation, he wasn't especially interested. His aim is to raise the school's students to higher forms of learning-analysis, synthe- sis, and application, as he put it. And he was concerned about his teachers, an energetic faculty with curricula and varied instructional methods he was loath to disrupt. On the other hand, the study's results could be instructive, and participa- tion would bring enticements in the form of smart boards and "clickers"-automated response systems for the classrooms of participating teachers. Money for new technology is fa- mously tight. A sixth grade social studies teacher, Patrice Bain, was eager to give it a try. For the researchers, a chance to work in the classroom was compelling, and the school's terms were ac- cepted: the study would be minimally intrusive by fitting within existing curricula, lesson plans, test formats, and teaching methods. The same textbooks would be used. The only differ- ence in the class would be the introduction of occasional short quizzes. The study would run for three semesters (a year and a half), through several chapters of the social studies textbook, covering topics such as ancient Egypt, Mesopotamia, India, and China. The project was launched in 2006. It would prove to be a good decision. For the six social studies classes a research assistant, Pooja Agarwal, designed a series of quizzes that would test students on roughly one-third of the material covered by the teacher. These quizzes were for no stakes," meaning that scores were not counted toward a grade. The teacher excused herself from the classroom for each quiz so as to remain unaware of which material was being tested. One quiz was given at the start of class, on material from assigned reading that hadn't yet been discussed. A second was given at the end of class after the teacher had covered the material for the day's lesson. And a review quiz was given twenty-four hours before each unit exam. There was concern that if students tested better in the final exam on material that had been quizzed than on material not quizzed, it could be argued that the simple act of reexposing them to the material in the quizzes was responsible for the superior learning, not the retrieval practice. To counter this possibility, some of the nonquizzed material was interspersed with the quiz material, provided as simple review statements, like "The Nile River has two major tributaries: the White Nile and the Blue Nile," with no retrieval required. The facts were quizzed for some classes but just restudied for others. The quizzes took only a few minutes of classroom time. After the teacher stepped out of the room, Agarwal projected a series of slides onto the board at the front of the room and read them to the students. Each slide presented either a mul- tiple choice question or a statement of fact. When the slide contained a question, students used clickers (handheld, cell- phone-like remotes) to indicate their answer choice: A, B, C, or D. When all had responded, the correct answer was revealed, so as to provide feedback and correct errors. (Although teachers were not present for these quizzes, under normal circumstances, with teachers administering quizzes, they would see immedi- ately how well students are tracking the study material and use the results to guide further discussion or study.) Unit exams were the normal pencil-and-paper tests given by the teacher. Exams were also given at the end of the se- mester and at the end of the year. Students had been exposed to all of the material tested in these exams through the teacher's normal classroom lessons, homework, worksheets, and so on, but they had also been quizzed three times on one-third of the material, and they had seen another third presented for additional study three times. The balance of the material was neither quizzed nor additionally reviewed in class beyond the initial lesson and whatever reading a student may have done. The results were compelling: The kids scored a full grade level higher on the material that had been quizzed than on the material that had not been quizzed. Moreover, test results for the material that had been reviewed as statements of fact but not quizzed were no better than those for the nonreviewed material. Again, mere rereading does not much help. In 2007, the research was extended to eighth grade science classes, covering genetics, evolution, and anatomy. The regi- men was the same, and the results equally impressive. At the end of three semesters, the eighth graders averaged 79 percent (C+) on the science material that had not been quizzed, com- pared to 92 percent (A-) on the material that had been quizzed. The testing effect persisted eight months later at the end- of-year exams, confirming what many laboratory studies have shown about the long-term benefits of retrieval practice. The effect doubtless would have been greater if the retrieval prac- tice had continued and occurred once a month, say, in the in- tervening months. The lesson from these studies has been taken to heart by many of the teachers at Columbia Middle School. Long after concluding their participation in the research studies, Patrice Bain's sixth grade social studies classes continue today to fol- low a schedule of quizzes before lessons, quizzes after lessons, and then a review quiz prior to the chapter test. Jon Wehren- berg, an eighth grade history teacher who was not part of the research, has knitted retrieval practice into his classroom in many different forms, including quizzing, and he provides ad- ditional online tools at his website, like flashcards and games. After reading passages on the history of slavery, for example, his students are asked to write down ten facts about slavery they hadn't known before reading the passages. You don't need electronic gadgetry to practice retrieval. Seven sixth and seventh graders needing to improve their reading and comprehension skills sat in Michelle Spivey's En- glish classroom one period recently with their reading books open to an amusing story. Each student was invited to read a paragraph aloud. Where a student stumbled, Miss Spivey had him try again. When he'd gotten it right, she probed the class to explain the meaning of the passage and what might have been going on in the characters' minds. Retrieval and elabora- tion; again, no technology required. Quizzes at Columbia Middle School are not onerous events. Following completion of the research studies, students' views were surveyed on this question. Sixty-four percent said the quizzing reduced their anxiety over unit exams, and 89 percent felt it increased learning. The kids expressed disappointment on days when clickers were not used, because the activity broke up the teacher's lecture and proved enjoyable. Principal Chamberlain, when asked what he thought the study results indicated, replied simply: "Retrieval practice has a significant impact on kids' learning. This is telling us that it's valuable, and that teachers are well advised to incorporate it into their instructional technique." Are similar effects found at a later age? Andrew Sobel teaches a class in international political eco- nomics at Washington University in St. Louis, a lecture course populated by 160-170 students, mostly freshmen and sopho- mores. Over a period of several years he noticed a growing problem with attendance. On any given day by midsemester, 25-35 percent of the class would be absent, compared to ear- lier in the semester when maybe 10 percent would be absent. The problem wasn't unique to his class, he says. A lot of pro- fessors give students their PowerPoint slides, so the students just stop coming to class. Sobel fought back by withholding his slides, but by the end of the semester, many students stopped showing up anyway. The class syllabus included two big tests, a midterm and a final. Looking for some way to leverage attendance, Sobel replaced the big tests with nine pop quizzes. Because the quizzes would determine the course grade and would be unannounced, students would be well advised to show up for class. The results were distressing. Over the semester, a third or more of the students bailed out. I really got hammered in the teaching reviews," Sobel told us. The kids hated it. If they didn't do well on a quiz they dropped the course rather than get a bad grade in it. Of those who stayed, I got this bifurcation between those who actually showed up and did the work, and those who didn't. I found myself handing out A-plusses, which I'd never given before, and more Cs than I'd ever given."10 With so much pushback, he had little choice but to drop the experiment and reinstate the old format, lectures with a mid- term and final. A couple of years later, however, after hearing a presentation about the learning benefits of testing, he added a third major test during the semester to see what effect it might have on his students learning. They did better, but not by as much as he'd hoped, and the attendance problems persisted. He scratched his head and changed the syllabus once again. This time he announced that there would be nine quizzes dur- ing the semester, and he was explicit about when they would be. No surprises, and no midterm or final exams, because he didn't want to give up that much of his lecture time. Despite fears that enrollments would plummet again, they actually increased by a handful. Unlike the pop quizzes, which kids hate, these were all on the syllabus. If they missed one it was their own fault. It wasn't because I surprised them or was being pernicious. They were comfortable with that." Sobel took satisfaction in seeing attendance improve as well. They would skip some classes on the days they didn't have a quiz, particu- larly the spring semester, but they showed up for the quizzes." Like the course, the quizzes were cumulative, and the ques- tions were similar to those on the exams he used to give, but the quality of the answers he was getting by midsemester was much better than he was accustomed to seeing on the mid- terms. Five years into this new format, he's sold on it. "The quality of discussions in class has gone way up. I see that big a difference in their written work, just by going from three exams to nine quizzes." By the end of the semester he has them writing paragraphs on the concepts covered in class, some- times a full-page essay, and the quality is comparable to what he's seeing in his upper division classes. "Anybody can design this structure. But I also realize that, Oh, god, if I'd done this years ago I would have taught them that much more stuff. The interesting thing about adopting this strategy is I now recognize that as good a teacher as I a might think I am, my teaching is only a component of their learning, and how I structure it has a lot to do with it, maybe even more." Meanwhile, the course enrollment has grown to 185 and counting Exploring Nuances Andy Sobel's example is anecdotal and likely reflects a variety of beneficial influences, not least being the cumulative learn- ing effects that accrue like compounded interest when course material is carried forward in a regime of quizzes across an entire semester. Nonetheless, his experience squares with em- pirical research designed to tease apart the effects and nu- ances of testing. For example, in one experiment college students studied prose passages on various scientific topics like those taught in college and then either took an immediate recall test after the initial exposure or restudied the material. After a delay of two days, the students who took the initial test recalled more of the material than those who simply restudied it (68 v. 54 per- cent), and this advantage was sustained a week later (56 v. 42 percent). Another experiment found that after one week a study-only group showed the most forgetting of what they ini- tially had been able to recall, forgetting 52 percent, compared to a repeated-testing group, who forgot only 10 percent." discoveries about how we learn motor tasks, like making lay- ups or driving a golf ball toward a distant green. In motor learning, trial and error with delayed feedback is a more awk- ward but effective way of acquiring a skill than trial and cor- rection through immediate feedback; immediate feedback is like the training wheels on a bicycle: the learner quickly comes to depend on the continued presence of the correction. In the case of learning motor skills, one theory holds that when there's immediate feedback it comes to be part of the task, so that later, in a real-world setting, its absence becomes a gap in the established pattern that disrupts performance. Another idea holds that frequent interruptions for feedback make the learning sessions too variable, preventing establish- ment of a stabilized pattern of performance.12 In the classroom, delayed feedback also yields better long- term learning than immediate feedback does. In the case of the students studying prose passages on science topics, some were shown the passage again even while they were asked to answer questions about it, in effect providing them with con- tinuous feedback during the test, analogous to an open-book exam. The other group took the test without the study mate- rial at hand and only afterward were given the passage and instructed to look over their responses. Of course, the open- book group performed best on the immediate test, but those who got corrective feedback after completing the test retained the learning better on a later test. Delayed feedback on writ- ten tests may help because it gives the student practice that's spaced out in time; as discussed in the next chapter, spacing practice improves retention.13 How does giving feedback on wrong answers to test questions affect learning? Studies show that giving feedback strengthens retention more than testing alone does, and, interestingly, some evidence shows that delaying the feedback briefly pro- duces better long-term learning than immediate feedback. This finding is counterintuitive but is consistent with researchers' Are some kinds of retrieval practice more effective for long- term learning than others? Tests that require the learner to retrieve material that is related but not tested. Further re- search is needed on this point, but it seems that retrieval prac- tice can make information more accessible when it is needed in various contexts. supply the answer, like an essay or short-answer test, or sim- ply practice with flashcards, appear to be more effective than simple recognition tests like multiple choice or true/false tests. However, even multiple choice tests like those used at Colum- bia Middle School can yield strong benefits. While any kind of retrieval practice generally benefits learning, the implication seems to be that where more cognitive effort is required for retrieval, greater retention results. Retrieval practice has been studied extensively in recent years, and an analysis of these studies shows that even a single test in a class can produce a large improvement in final exam scores, and gains in learning continue to increase as the number of tests increases.14 Whichever theories science eventually tells us are correct about how repeated retrieval strengthens memory, empirical research shows us that the testing effect is realthat the act of retrieving a memory changes the memory, making it easier to retrieve again later. Do students resist testing as a tool for learning? Students do generally dislike the idea of tests, and it's not hard to see why, in particular in the case of high-stakes tests like midterms and finals, where the score comes with significant consequences. Yet in all studies of testing that reported students' attitudes, the students who were tested frequently rated their classes more favorably at the end of the semester than those tested less frequently. Those who were frequently tested reached the end of the semester on top of the material and did not need to cram for exams. How widely is retrieval practice used as a study technique? In one survey, college students were largely unaware of its effec- tiveness. In another survey, only 11 percent of college students said they use this study strategy. Even when they did report testing themselves, they mostly said they did it to discover what they didn't know, so they could study that material more. That's a perfectly valid use of testing, but few students realize that retrieval itself creates greater retention.15 How does taking a test affect subsequent studying? After a test, students spend more time restudying the material they missed, and they learn more from it than do their peers who restudy the material without having been tested. Students whose study strategies emphasize rereading but not self-testing show over- confidence in their mastery. Students who have been quizzed have a double advantage over those who have not: a more accurate sense of what they know and don't know, and the strengthening of learning that accrues from retrieval practice. 16 Is repeated testing simply a way to expedite rote learning? In fact, research indicates that testing, compared to rereading, can facilitate better transfer of knowledge to new contexts and problems, and that it improves one's ability to retain and Are there any further, indirect benefits of regular, low-stakes classroom testing? Besides strengthening learning and reten- tion, a regime of this kind of testing improves student atten- dance. It increases studying before class (because students know they'll be quizzed), increases attentiveness during class if students are tested at the end of class, and enables students to better calibrate what they know and where they need to bone up. It's an antidote to mistaking fluency with the text, resulting from repeated readings, for mastery of the subject. Frequent low-stakes testing helps dial down test anxiety among students by diversifying the consequences over a much larger sample: no single test is a make-or-break event. And this kind of testing enables instructors to identify gaps in stu- dents' understanding and adapt their instruction to fill them. These benefits of low-stakes testing accrue whether instruc- tion is delivered online or in the classroom.17 While cramming can produce better scores on an immedi- ate exam, the advantage quickly fades because there is much greater forgetting after rereading than after retrieval practice. The benefits of retrieval practice are long-term. Simply including one test (retrieval practice) in a class yields a large improvement in final exam scores, and gains continue to increase as the frequency of classroom testing increases. Testing doesn't need to be initiated by the instructor. Stu- dents can practice retrieval anywhere; no quizzes in the class- room are necessary. Think flashcardsthe way second grad- ers learn the multiplication tables can work just as well for learners at any age to quiz themselves on anatomy, mathemat- ics, or law. Self-testing may be unappealing because it takes more effort than rereading, but as noted already, the greater the effort at retrieval, the more will be retained. Students who take practice tests have a better grasp of their progress than those who simply reread the material. Similarly, such testing enables an instructor to spot gaps and miscon- ceptions and adapt instruction to correct them. Giving students corrective feedback after tests keeps them from incorrectly retaining material they have misunderstood and produces better learning of the correct answers. Students in classes that incorporate low-stakes quizzing come to embrace the practice. Students who are tested fre- quently rate their classes more favorably. The Takeaway Practice at retrieving new knowledge or skill from memory is a potent tool for learning and durable retention. This is true for anything the brain is asked to remember and call up again in the future-facts, complex concepts, problem-solving tech- niques, motor skills. Effortful retrieval makes for stronger learning and reten- tion. We're easily seduced into believing that learning is better when it's easier, but the research shows the opposite: when the mind has to work, learning sticks better. The greater the effort to retrieve learning, provided that you succeed, the more that learning is strengthened by retrieval. After an initial test, delaying subsequent retrieval practice is more potent for rein- forcing retention than immediate practice, because delayed retrieval requires more effort. Repeated retrieval not only makes memories more durable but produces knowledge that can be retrieved more readily, in more varied settings, and applied to a wider variety of problems. What about Principal Roger Chamberlain's initial concerns about practice quizzing at Columbia Middle Schoolthat it might be nothing more than a glorified path to rote learning? When we asked him this question after the study was com- pleted, he paused for a moment to gather his thoughts. What I've really gained a comfort level with is this: for kids to be able to evaluate, synthesize, and apply a concept in different settings, they're going to be much more efficient at getting there when they have the base of knowledge and the reten- tion, so they're not wasting time trying to go back and figure out what that word might mean or what that concept was about. It allows them to go to a higher level. 3 Mix Up Your Practice IT MAY NOT BE INTUITIVE that retrieval practice is a more powerful learning strategy than repeated review and rereading, yet most of us take for granted the importance of testing in sports. It's what we call practice- practice-practice." Well, here's a study that may surprise you. A group of eight-year-olds practiced tossing beanbags into buckets in gym class. Half of the kids tossed into a bucket three feet away. The other half mixed it up by tossing into buckets two feet and four feet away. After twelve weeks of this they were all tested on tossing into a three-foot bucket. The kids who did the best by far were those who'd practiced on two- and four-foot buckets but never on three-foot buckets.' Why is this? We will come back to the beanbags, but first a little insight into a widely held myth about how we learn. Almost everywhere you look, you find examples of massed practice: summer language boot camps, colleges that offer con- centration in a single subject with the promise of fast learning, continuing education seminars for professionals where train- ing is condensed into a single weekend. Cramming for exams is a form of massed practice. It feels like a productive strategy, and it may get you through the next day's midterm, but most of the material will be long forgotten by the time you sit down for the final. Spacing out your practice feels less productive for the very reason that some forgetting has set in and you've got to work harder to recall the concepts. It doesn't feel like you're on top of it. What you don't sense in the moment is that this added effort is making the learning stronger.2 The Myth of Massed Practice Most of us believe that learning is better when you go at something with single-minded purpose: the practice-practice- practice that's supposed to burn a skill into memory. Faith in focused, repetitive practice of one thing at a time until we've got it nailed is pervasive among classroom teachers, athletes, corporate trainers, and students. Researchers call this kind of practice "massed," and our faith rests in large part on the simple fact that when we do it, we can see it making a differ- ence. Nevertheless, despite what our eyes tell us, this faith is misplaced. If learning can be defined as picking up new knowledge or skills and being able to apply them later, then how quickly you pick something up is only part of the story. Is it still there when you need to use it out in the everyday world? While practicing is vital to learning and memory, studies have shown that practice is far more effective when it's broken into sepa- rate periods of training that are spaced out. The rapid gains produced by massed practice are often evident, but the rapid forgetting that follows is not. Practice that's spaced out, inter- leaved with other learning, and varied produces better mas- tery, longer retention, and more versatility. But these benefits come at a price: when practice is spaced, interleaved, and varied, it requires more effort. You feel the increased effort, but not the benefits the effort produces. Learning feels slower from this kind of practice, and you don't get the rapid im- provements and affirmations you're accustomed to seeing from massed practice. Even in studies where the participants have shown superior results from spaced learning, they don't perceive the improvement; they believe they learned better on the material where practice was massed. Spaced Practice The benefits of spacing out practice sessions are long estab- lished, but for a vivid example consider this study of thirty- eight surgical residents. They took a series of four short lessons in microsurgery: how to reattach tiny vessels. Each lesson included some instruction followed by some prac- tice. Half the docs completed all four lessons in a single day, which is the normal in-service schedule. The others completed the same four lessons but with a week's interval between them.3 In a test given a month after their last session, those whose lessons had been spaced a week apart outperformed their col- leagues in all areas-elapsed time to complete a surgery, num- ber of hand movements, and success at reattaching the sev- ered, pulsating aortas of live rats. The difference in performance between the two groups was impressive. The residents who had taken all four sessions in a single day not only scored lower on all measures, but 16 percent of them damaged the rats' vessels beyond repair and were unable to complete their surgeries. Why is spaced practice more effective than massed practice? It appears that embedding new learning in long-term memory requires a process of consolidation, in which memory traces (the brain's representations of the new learning) are strength- ened, given meaning, and connected to prior knowledgea process that unfolds over hours and may take several days. Rapid-fire practice leans on short-term memory. Durable learning, however, requires time for mental rehearsal and the other processes of consolidation. Hence, spaced practice works better. The increased effort required to retrieve the learning after a little forgetting has the effect of retriggering consoli- dation, further strengthening memory. We explore some of the theories about this process in the next chapter. final test a week later, the students who had practiced solving problems clustered by type averaged only 20 percent correct, while the students whose practice was interleaved averaged 63 percent. The mixing of problem types, which boosted final test performance by a remarkable 215 percent, actually im- peded performance during initial learning. 4 Now, suppose you're a trainer in a company trying to teach employees a complicated new process that involves ten proce- dures. The typical way of doing this is to train up in proce- dure 1, repeating it many times until the trainees really seem to have it down cold. Then you go to procedure 2, you do many repetitions of 2, you get that down, and so on. That appears to produce fast learning. What would interleaved practice look like? You practice procedure 1 just a few times, then switch to procedure 4, then switch to 3, then to 7, and so on. (Chapter 8 tells how Farmers Insurance trains new agents in a spiraling series of exercises that cycle back to key skillsets in a seem- ingly random sequence that adds layers of context and mean- ing at each turn.) The learning from interleaved practice feels slower than learning from massed practice. Teachers and students sense the difference. They can see that their grasp of each element is coming more slowly, and the compensating long-term advan- tage is not apparent to them. As a result, interleaving is unpop- ular and seldom used. Teachers dislike it because it feels slug- gish. Students find it confusing: they're just starting to get a handle on new material and don't feel on top of it yet when they are forced to switch. But the research shows unequivo- cally that mastery and long-term retention are much better if you interleave practice than if you mass it. Interleaved Practice Interleaving the practice of two or more subjects or skills is also a more potent alternative to massed practice, and here's a quick example of that. Two groups of college students were taught how to find the volumes of four obscure geometric solids (wedge, spheroid, spherical cone, and half cone). One group then worked a set of practice problems that were clus- tered by problem type (practice four problems for computing the volume of a wedge, then four problems for a spheroid, etc.). The other group worked the same practice problems, but the sequence was mixed (interleaved) rather than clustered by type of problem. Given what we've already presented, the results may not surprise you. During practice, the students who worked the problems in clusters (that is, massed) averaged 89 percent correct, compared to only 60 percent for those who worked the problems in a mixed sequence. But in the Varied Practice Okay, what about the beanbag study where the kids who did best had never practiced the three-foot toss that the other kids had only practiced? The beanbag study focused on mastery of motor skills, but much evidence has shown that the underlying principle ap- plies to cognitive learning as well. The basic idea is that varied practicelike tossing your beanbags into baskets at mixed distancesimproves your ability to transfer learning from one situation and apply it successfully to another. You develop a broader understanding of the relationships between different conditions and the movements required to succeed in them; you discern context better and develop a more flexible move- ment vocabulary" different movements for different situa- tions. Whether the scope of variable training (e.g., the two- and four-foot tosses) must encompass the particular task (the three-foot toss) is subject for further study. The evidence favoring variable training has been supported by recent neuroimaging studies that suggest that different kinds of practice engage different parts of the brain. The learning of motor skills from varied practice, which is more cognitively challenging than massed practice, appears to be consolidated in an area of the brain associated with the more difficult pro- cess of learning higher-order motor skills. The learning of mo- tor skills from massed practice, on the other hand, appears to be consolidated in a different area of the brain that is used for learning more cognitively simple and less challenging motor skills. The inference is that learning gained through the less challenging, massed form of practice is encoded in a simpler or comparatively impoverished representation than the learn- ing gained from the varied and more challenging practice which demands more brain power and encodes the learning in a more flexible representation that can be applied more broadly. Among athletes, massed practice has long been the rule: take your hook shot, knock the twenty-foot putt, work your backhand return, throw the pass while rolling out: again and again and again to get it right and train your muscle mem- ory." Or so the notion holds. The benefits of variable training for motor learning have been gaining broader acceptance, albeit slowly. Consider the one-touch pass in hockey. That's where you receive the puck and immediately pass it to a team- mate who's moving down the ice, keeping the opposition off balance and unable to put pressure on the puck carrier. Jamie Kompon, when he was assistant coach of the Los Angeles Kings, was in the habit of running team practice on one-touch passes from the same position on the rink. Even if this move is interleaved with a sequence of other moves in practice, if you only do it at the same place on the rink or in the same sequence of moves, you are only, as it were, throwing your beanbags into the three-foot bucket. Kompon is onto the difference now and has changed up his drills. Since we talked, he's gone over to the Chicago Blackhawks. We would have said Keep an eye on those Blackhawks" here, but as we revise to go into produc- tion, Kompon and team have already won the Stanley Cup. Perhaps no coincidence? The benefits of variable practice for cognitive as opposed to motor skills learning were shown in a recent experiment that adapted the beanbag test to verbal learning: in this case, the students solved anagrams-that is, they rearranged letters to form words (tmoce becomes comet). Some subjects practiced the same anagram over and over, whereas others practiced mul- tiple anagrams for the word. When they were all tested on the same anagram that the former group had practiced on, the latter group performed better on it! The same benefits will apply whether you are practicing to identify tree species, differentiate the principles of case law, or master a new com- puter program. Developing Discrimination Skills Compared to massed practice, a significant advantage of in- terleaving and variation is that they help us learn better how to assess context and discriminate between problems, selecting and applying the correct solution from a range of possibilities. In math education, massing is embedded in the textbook: each chapter is dedicated to a particular kind of problem, which you study in class and then practice by working, say, twenty ex- amples for homework before you move on. The next chapter has a different type of problem, and you dive into the same kind of concentrated learning and practice of that solution. On you march, chapter by chapter, through the semester. But then, on the final exam, lo and behold, the problems are all mixed up: you're staring at each one in turn, asking yourself Which algorithm do I use? Was it in chapter 5, 6, or 7? When you have learned under conditions of massed or blocked repe- tition, you have had no practice on that critical sorting process. But this is the way life usually unfolds: problems and oppor- tunities come at us unpredictably, out of sequence. For our learning to have practical value, we must be adept at discerning "What kind of problem is this?" so we can select and apply an appropriate solution. Several studies have demonstrated the improved powers of discrimination to be gained through interleaved and varied practice. One study involved learning to attribute paintings to the artists who created them, and another focused on learning to identify and classify birds. Researchers initially predicted that massed practice in identifying painters' works (that is, studying many examples of one painter's works before moving on to study many ex- amples of another's works) would best help students learn the defining characteristics of each artist's style. Massed practice of each artist's works, one artist at a time, would better enable students to match artworks to artists later, compared to inter- leaved exposure to the works of different artists. The idea was that interleaving would be too hard and confusing; students would never be able to sort out the relevant dimensions. The researchers were wrong. The commonalities among one paint- er's works that the students learned through massed practice proved less useful than the differences between the works of multiple painters that the students learned through interleav- ing. Interleaving enabled better discrimination and produced better scores on a later test that required matching the works with their painters. The interleaving group was also better able to match painters' names correctly to new examples of their work that the group had never viewed during the learn- ing phase. Despite these results, the students who participated in these experiments persisted in preferring massed practice, convinced that it served them better. Even after they took the test and could have realized from their own performance that interleaving was the better strategy for learning, they clung to their belief that the concentrated viewing of paintings by one artist was better. The myths of massed practice are hard to ex- orcise, even when you're experiencing the evidence yourself.7 The power of interleaving practice to improve discrim- inability has been reaffirmed in studies of people learning bird classification. The challenge here is more complex than it might seem. One study addressed twenty different bird fami- lies (thrashers, swallows, wrens, finches, and so on). Within each family, students were presented with a dozen species Improving Complex Mastery for Medical Students (brown thrasher, curve-billed thrasher, Bendire's thrasher, etc.). To identify a bird's family, you consider a wide range of traits like size, plumage, behavior, location, beak shape, iris color, and so on. A problem in bird identification is that mem- bers of a family share many traits in common but not all. For instance, many but not all thrashers have a long, slightly hooked beak. There are traits that are typical of a family but none that occur in all members of that family and can serve as unique identifiers. Because rules for classification can only rely on these characteristic traits rather than on defining traits (ones that hold for every member), bird classification is a mat- ter of learning concepts and making judgments, not simply memorizing features. Interleaved and variable practice proved more helpful than massed practice for learning the underlying concepts that unite and differentiate the species and families. To paraphrase a conclusion from one of these studies, re- call and recognition require "factual knowledge," considered to be a lower level of learning than "conceptual knowledge." Conceptual knowledge requires an understanding of the in- terrelationships of the basic elements within a larger structure that enable them to function together. Conceptual knowledge is required for classification. Following this logic, some people argue that practicing retrieval of facts and exemplars would fall short as a strategy for comprehending general characteris- tics that are required for higher levels of intellectual behavior. The bird classification studies suggest the opposite: strategies of learning that help students identify and discern complex prototypes (family resemblances) can help them grasp the kinds of contextual and functional differences that go beyond the acquisition of simple forms of knowledge and reach into the higher sphere of comprehension. The distinction between straightforward knowledge of facts and deeper learning that permits flexible use of the knowledge may be a little fuzzy, but it resonates with Douglas Larsen at Washington University Medical School in St. Louis, who says that the skills required for bird classification are similar to those required of a doctor diagnosing what's wrong with a patient. The reason variety is important is it helps us see more nuances in the things that we can compare against," he says. That comes up a lot in medicine, in the sense that every patient visit is a test. There are many layers of explicit and implicit memory involved in the ability to discriminate be- tween symptoms and their interrelationships. Implicit mem- ory is your automatic retrieval of past experience in interpret- ing a new one. For example, the patient comes in and gives you a story. As you listen, you're consciously thinking through your mental library to see what fits, while also unconsciously polling your past experiences to help interpret what the pa- tient is telling you. Then you're left with making a judgment call," Larsen says. Larsen is a pediatric neurologist seeing patients in the uni- versity clinic and hospital. He's a busy guy: in addition to practicing medicine, he supervises the work of physicians in training, he teaches, and as time permits, he conducts research into medical education, working in collaboration with cogni- tive psychologists. He's drawing on all of these roles to redesign and strengthen the school's training curriculum in pediatric neurology. As you'd expect, the medical school employs a wide spec- trum of instructional techniques. Besides classroom lectures and labs, students practice resuscitations and other procedures on high-tech mannequins in three simulation centers the school maintains. Each patient" is hooked up to monitors, has a heartbeat, blood pressure, pupils that dilate and constrict, and the ability to listen and speak, thanks to a controller who observes and operates the mannequin from a back room. The school also makes use of "standardized patients," actors who follow scripts and exhibit symptoms the students are required to diagnose. The center is set up like a regular medical clinic, and students must show proficiency in all aspects of a patient encounter, from bedside manner, physical exam skills, and re- membering to ask the full spectrum of pertinent questions to arriving at a diagnosis and treatment plan. From studies of these teaching methods, Larsen has drawn some interesting conclusions. Firstand this may seem self- evident: you do better on a test to demonstrate your compe- tency at seeing patients in a clinic if your learning experience has involved seeing patients in a clinic. Simply reading about patients is not enough. However, on written final exams, medi- cal students who have examined patients and those who have learned via written tests do equally well. The reason is that in a written test the student is being given considerable structure and being asked for specific information. When examining the patient, you have to come up on your own with the right mental model and the steps to follow. Having practiced these steps on patients or simulated patients improves performance relative to just reading about how to do it. In other words, the kind of retrieval practice that proves most effective is one that reflects what you'll be doing with the knowledge later. It's not just what you know, but how you practice what you know that determines how well the learning serves you later. As the sports adage goes, practice like you play and you will play like you practice." This conclusion lines up with other research into learning, and with some of the more sophisticated training practices in science and industry, including the increasingly broad use of si