Future Fellows - June 2006
Back to Main Page

How Exam Pass Marks Are Set

Recently, the SOA Board of Governors and the CAS Board of Directors each held discussions on pass mark setting for multiple-choice exams. Each Board directed its respective Examination Committees to issue a public statement describing content-based pass marks with the intent of enhancing candidates’ understanding of how pass marks are set for multiple-choice exams. The following was posted in the “Admissions” section of the CAS Web Site in April 2006.
Person Studying

Why Content-Based Pass Marks?

The goal of the examination systems of both the SOA and CAS is to pass all candidates who have demonstrated adequate knowledge of the syllabus material and to fail those candidates who have not. The objective of the examinations is to evaluate candidate performance using criteria for demonstrating adequate knowledge that remain constant throughout the lifetime of the exam series. Pre-set pass marks (e.g., a candidate will pass if he/she answers x% of the questions correctly) are counter to this philosophy. The examinations are meant to measure the candidate’s level of achievement of the required learning objectives and their required level of capability of accomplishing specified learning outcomes.

Multiple-Choice Pass Mark Setting

Exam 1/P is administered and scored according to computer-based testing methodologies. For the other multiple-choice examinations, a modified Angoff passing score study is performed. This is a common testing and measurement technique where a panel of experts in the subject material reviews the examination. Each expert is asked to review each question in the examination and assess the difficulty of that question. More specifically, they are asked to estimate the likelihood that a candidate with minimum adequate knowledge competency would answer the question correctly. The sum of these probabilities, averaged across the panel of experts, gives a preliminary estimate of the pass mark.

The estimated pass mark resulting from the modified Angoff passing score study is compared to and balanced with the actual performance statistics on the examination in finalizing the pass mark. The effects of any particularly difficult questions are also factored into the determination of the final pass mark.

CAS Written-Answer Pass Mark Setting

For CAS examinations consisting in whole or in part of written-answer questions, the assessment process is somewhat different. Before the exam is administered, a pass mark panel reviews the exam and assesses it based on how the panelists think a minimally qualified candidate will perform based on a predetermined definition of the minimally qualified candidate. This process follows the same basic technique used for multiple choice exams. Based on this assessment, an expected pass mark is set.

Following the administration of each exam, each answer is graded simultaneously by two graders who must reconcile their techniques and grades. When all questions have been scored, the committee chooses a preliminary pass mark based on the results of the pass mark panel augmented by actual performance of the current candidates versus historical performance of previous candidate cohorts. Candidate papers with scores close to the preliminary pass park are regraded to ensure correct and consistent scoring. The Examination Committee then determines the final pass mark by again balancing actual performance statistics against minimum adequate knowledge while taking into account other factors such as time pressure situations that may have occurred on some questions. The effects of any particularly difficult questions are also factored in when determining the final pass mark.

Who Decides the Final Pass Marks?

With the use of content-based pass marks, fluctuation in the pass rate from session to session is expected. A recommended pass mark is reached by consultation between the Part Chairperson and the Examination Committee Chairperson. Any significant deviations from the a priori pass mark set by the pass mark panel are explored at this time. The recommended pass mark and explanations for deviations from the a priori pass mark and any abnormal passing percentages are submitted to the Vice President-Admissions who approves the final pass mark. Upon approval by the Vice President-Admissions, the final pass mark and exam statistics are forwarded to the Executive Council.

© 2014 Casualty Actuarial Society. All Rights Reserved.