Retention Analytics

Retention is the most important aspect of learning.  I mean, learning without retention isn't learning at all.  You have to be able to recall information later in order to utilize your education.  The concept of "memory" requires encoding, storage, AND retrieval.  Here is a wikipedia article on the "forgetting curve."

I posted thoughts a while ago about how important reviewing is to long-term memory.  This is somewhat of a follow-up.

In 8th-grade medieval history, we switch to literature in the third quarter.  All the sections read The Song of Roland, which is a medieval era Frankish tale of deeds, or "chanson de geste". Basically an epic poem, in medieval French. There are certainly historical aspects, as the poem tells the tale of Charlemagne campaigning against Spain and its recently-established Muslim kingdoms.    Most of the information, however, is composed of legends and embellishments.

The point here is that my classes haven't focused on real medieval history since December.  Close to three months.  We've been reading and discussing it as literature, with very few references to previous material.  We finished the Song of Roland Friday, and we have one week until spring break.  My goal for this week was to re-familiarize my students with things they had already learned in the fall.

I had a few review activities planned today, but this morning I was struck with a brilliant, savant-esque idea — have them re-take the fall semester final exam.

This is the type of idea that, the first few times you consider it, sounds as if it won't work.  Then, when you really think about it, it's perfect.

One aspect of the perfection was seeing their faces when I broke the news at the beginning of class.  It was a fluid shift from horror (I'm gonna fail!) to resignation (He's not joking this time again!) to resolve (I'm going to FOCUS) to anticipation (It won't count against my grade!) to, in a few cases, confidence (I remember this!).  Some of them clearly relished the opportunity to test their knowledge, no strings attached.

My semester final was half-discussion, half-objective questions.  The objective questions included multiple choice, a timeline, and a map.  Given my time constraints, I decided to give them the 40 multiple choice questions in class.  That way, they would have time to trade n' grade in class.

While they answered the questions, I went through and added up how many M/C questions each of them missed the first time around.  Here are some numbers:

22 students in the class (I only did it in one section).
Final exam average overall: 86.1%
Missed 8.318 questions out of 40 M/C, on average.
Average on M/C questions portion: 79.2%

Now, I'm notorious, perhaps even infamous, for having difficult multiple choice questions.  I don't follow the unwritten rules.  I have options like "All of the below", or give eight possible answers instead of four, or word phrases confusingly.  Not only that, but you really have to understand the information to get the questions right (at a passable rate).  I like to think I've instilled a fear in my students similar to what Dr. Pongracic does to his economics students at Hillsdale.  Before taking his classes, I never in my life had struggled on M/C tests, and I wouldn't hesitate to take M/C over anything else, anytime.  After Pongracic, I viewed multiple choice in a different way, almost like they had betrayed me and my capability to dominate standardized testing.

Here's an example of one of my multiple choice questions I consider to be in the "comprehensive" category.

"What event did NOT assist the spread of feudalism after the 9th century?
a) the Treaty of Verdun
b) the Norman Conquest
c) The Capetian Dynasty
d) the fall of the Carolingian Empire"

In order to get this question right, you need to have a decent grasp of several key concepts.  You need to know each of the possible answers, when/where they occurred what effect they had immediately and afterward.  For example, the Norman Conquest effect wasn't obvious.  William the Conqueror took England, but wanted it primarily for tax purposes, so he brought feudalism over from the mainland so England could operate in his absence.

I also like it because you could arrive at the correct answer by process of elimination — if you know enough about the "wrong" answers, you can identify the correct one.  The Treaty of Verdun is very much connected to the fall of the Carolingian Empire, so if you know that you can eliminate both answers.  Additionally, if you truly understand how feudalism operates, you will understand that a dynasty means a strong central ruler, which feudalism typically did not have.

Or, you have at least a 25% chance of guessing right. [The answer is "c"]

This demonstrates two things at least.  1. Multiple choice can be an effective method of testing in certain cases, with the right approach.  2. I can overanalyze anything.  Well, almost anything.  Most things.  I overanalyze every time, 60% of the time.

Okay, they just finished their re-examination, we can look at the data!  My new thing as of yesterday is fake-live-blogging events that I shouldn't be.

This is a big moment.  How bad was it going to be?  As a constant crammer through high school, my expectations are probably lower than most...but this isn't just a FASCINATING experiment, it also reflects on my ability to teach. gulp.

Here's what I'll do.  I'll give you the number MISSED by each student the first time, most to least.

23, 18, 16, 13, 12, 12, 11, 10, 9, 9, 8, 8, 7, 6, 5, 4, 4, 3, 2, 2, 1, 0

Annnnd the second time around, three months later.  Order is matched from the first list.

25, 16, 20, 14, 22, 19, 13, 14, 9, 11, 10, 16, 10, 11, 7, 6, 7, 12, 1, 4, 4, 1

The difference comes out to be -3.13 per student, so 11.454 missed questions, or 71.36%.  (down from 8.318 and 79.2%, remember?)

This is the part where I get over my head, statistically speaking.  I could go into statistical regression analysis and the guessing-odds, but the pool is just too small to spend time doing that.

That being said, there are definitely a few interesting cases in those 22.  Notably, the 12-to-22, 3-to-12, and 8-to-16.  Then there's the 18-to-16.  The most obvious explanations are a few successful crammers and a lucky guesser, but I'd love to do this a few more times with a few other classes (or, even better, the same class) and see what happens.

Even though there's waay too many qualifiers and variables to mention, for now, I'm going to voice my gut instinct and say this is a positive outcome.  Like I told the kids, they lost about one question-worth of knowledge per month, on average.  If they had performed that way on the first final, the test average would have been about an B-.  More importantly, I don't prioritize memorizing a ton of stock information in my classes.  I try to focus on broad ideas, recognizing cause and analyzing effect, and trends.  If they maintained most of the specific information, I would hope they've retained even more of the broad concepts (the stuff they don't realize they are learning).

I bet you didn't think I could write this much about a silly re-taken test.