Setting the Bar for the Minimally Qualified Candidate

One of the things I like most about the actuarial profession is how collegial it is. Actuaries study together and try to help each other, and I like to think that that's partially because of the exam process. Since exams aren't graded on a curve, we know it doesn't hurt us if our colleagues do well on an exam (and it doesn't help us if they do poorly, either). The only person we really have to beat on an exam is the Minimally Qualified Candidate. The Minimally Qualified Candidate (MQC, for short) isn't a real person — it represents what a theoretical person who's just qualified to pass would be able to do, and it's used to set the pass mark. I talked to one of the Exam Committee's General Officers to figure out what goes into determining this MQC standard for each exam, and found out that it's more complicated than you may think.

Descriptions of what the MQC should understand and be able to do under exam conditions — we'll call these MQC Narratives — are written for each knowledge statement and learning objective. MQC Narratives are based on what actuaries do in practice. For example, the standard may be higher for doing a Bornhuetter-Ferguson problem than for a Cape Cod problem, since the BF method is more commonly used than the Cape Cod method. (This is my example. Please don't take this as a suggestion to ignore the Cape Cod method — anything on the syllabus is of course totally testable, and I don't want to be responsible for any lost points!) The MQC Narratives are set when material is added to or removed from the syllabus, and they're reviewed and may be tweaked before each sitting of the exam.

Once an exam has been written, a panel (aptly named the Pass Mark Panel) gets together to spend a full day discussing the exam and coming to a consensus on what the pass mark should be for each question. The panel typically consists of one or two General Officers of the Exam Committee, the Part Chair, the Vice Chair of Grading, Vice Chair of Syllabus, a few current graders/writers of the exam, and two pre-testers. The panel members all review the MQC Narratives before the meeting and come with initial estimates of the pass mark by question. The panel then discusses each question, covering things like the level of difficulty, familiarity of the question, complexity of any calculations (including possible calculation error), partial credit recommendations, and how much synthesis the question requires. After all this deliberation, a preliminary pass mark is set.

Then, after we take the exam and hope we earned enough points to pass, the Exam Committee gets to work on grading and finalizing the pass mark. Each question has at least two graders, and in addition to coming to an agreement on the score for each candidate's answer, they have to come to an agreement on what the MQC score should be. They don't see what the Pass Mark Panel said, but they base their estimate on the same MQC Narratives and partial credit guide that the Pass Mark Panel used, only adjusted to reflect any changes the graders might make to the partial credit guide as sometimes graders would expand the partial credit guide to encompass additional answers as they go through the actual answers submitted. They also take into consideration how leniently (or harshly) they graded answers relative to what the Pass Mark Panel assumed. Any differences between the Pass Mark Panel's initial estimate and the graders' estimate are reconciled, with a single MQC score selected for each question. The sum of those selected MQC scores becomes the tentative pass mark for the exam as a whole, adjusted only in unusual circumstances (such as when an exam is considered to be unusually long).

I was very curious about how candidate responses factored into setting the MQC, so I asked the General Officer about it. He said that candidate responses were often used as a metric for how clearly the exam was written and how comprehensive the initial answer set was, as opposed to being a direct input to the MQC score. So, for example, if most candidates score really well on a question because they all genuinely knew the material well, the MQC would not be adjusted. However, if there's a question where initially the candidate had to give three of four possible answers (and the Pass Mark Panel was only familiar with these four possible answers), but during grading the graders determined that there were 12 possible answers, the MQC may be adjusted since remembering three out of 12 options is easier than remembering three out of four. The percent of people who pass is a similarly used metric. If candidates performed much differently than expected, the committee tries to understand what's really driving it and would discuss the MQC scores for certain questions in greater detail to ensure the selection is reasonable.

I was also curious about what happens when a question is defective. It's most helpful when defective questions are identified before grading (through emails to defective-item@casact.org), because it's much easier to deal with them at the point of grading than if they're identified later in the process. When a question is genuinely defective but in a way that candidates are still able to demonstrate knowledge on the topic, some partial credit is often given to candidates who attempt it and the exam is graded twice — once with the MQC for that question set to zero and no credit given, and once with an MQC set for the problem (with the understanding that the question was at least partially defective) and with partial credit given. If a candidate passes under either scenario, then they pass the exam! This removes any penalty for someone who skipped it, but rewards candidates who demonstrated knowledge of the topic.

Although overall difficulty of an exam can change from sitting to sitting, the Exam Committee does everything it can to make the standard to pass the exam consistent over time using the MQC. More information about the pass mark setting process can be found here.

So good luck in your studying, and may you be more qualified than the Minimally Qualified Candidate!