Actuarial Review Return to Main Page

Chauncey Project To Improve Admission Processes

by Thomas G. Myers, CAS Vice President-Admissions

In 1998, the CAS Board created the Task Force on the Review of Education and Examination Process and Procedures to undertake a thorough review of existing admissions processes and provide recommendations for ways to improve those processes. In November 1999, the task force issued its final report that included a recommendation to seek assistance from professional consultants to assist in improving the CAS education and examination processes.

The CAS leadership agreed with the task force recommendation to seek external assistance from someone with extensive experience in the field of professional education. In fall 2000, the CAS asked for assistance with a major improvement initiative. The CAS received four proposals and ultimately chose The Chauncey Group International, a subsidiary of Educational Testing Services, for the project.

In spring 2001, consultants from The Chauncey Group conducted an audit of CAS admissions processes and reported on how CAS processes compared to their standards for professional education programs (www.casact.com/admissions/reports/2002audit.pdf). In their report, Chauncey's consultants identified a number of things that the CAS does very well:

Chauncey's consultants also identified several opportunities for improvement. The Chauncey Group recommended that the CAS should:

Writing Better Learning Objectives
In late 2001, Chauncey met with syllabus and exam experts for each exam to work on writing better learning objectives. The key issue in writing good learning objectives is to identify what successful candidates should be able to do following completion of the learning experience and not just what knowledge they should gain. Supporting each learning objective is a series of items candidates need to know (knowledge statements) and the readings that provide the appropriate learning material.

At each meeting, attendees started from scratch in identifying what successful candidates should be able to do following completion of each exam unit. They also identified the supporting knowledge statements and readings for each learning objective. In some cases, they identified desirable learning objectives for which the current syllabus readings did not provide adequate learning material. This helped to distinguish and prioritize areas for improving the current reading list. In cases where adequate readings were not readily available, learning objectives needed to be classified for potential future use but excluded from the current objective list.

Following each meeting, the draft learning objectives were extensively reviewed within the Syllabus and Examination Committees. In spring 2003, the CAS Executive Council approved the final learning objectives, including approximate weights to guide candidates on the relative importance of each learning objective. The new learning objectives for Fall 2003 Exams were published in June. The new learning objectives will be published in the Syllabus of Examinations and will be reviewed annually by the Syllabus committee and updated as appropriate.

Writing Better Exam Questions
In the past, the CAS has taken capable actuaries and asked them to write exam questions with relatively little training. "Training" involved providing copies of past exams to use as a model, which simply perpetuated any bad habits that question writers were using.

With oversight from officers of the Examination Committee, The Chauncey Group developed a tailored training program for CAS question writers that is intended to convey the principles of good question writing and reinforce the need to link the writing of questions to the intended learning objectives for each exam. Chauncey's consultants again emphasized that good learning objectives focus on what successful candidates should be able to do and that good questions should follow a similar orientation—they should test whether candidates are capable of completing the relevant actions, and not just what candidates know.

The new one-day training program gives attendees an opportunity to apply the principles of good question writing. Each participant writes 3-4 questions of various types. These questions are then shared, and the entire group is asked to critique each question with input from the experienced Chauncey consultant. During the course of the day, attendees discuss 50-70 different questions and see real examples of what differentiates good from poor exam questions. Feedback from the initial sessions indicated that attendees found this new training program to be extremely valuable.

The CAS piloted this new training program in spring 2002 for the Fall 2002 Exam Committees. Initial training of all non-joint exam committees was completed by spring 2003. In the future, the CAS will continue to offer regular training workshops and require all new question writers to attend training.

Along with the training and the learning objective initiatives, the Exam Committees have also changed how question-writing assignments are made. In the past, question writers were asked to write a certain number of questions from particular syllabus readings without any guidance about where to focus. This often led to questions that simply tested whether candidates had memorized a particular portion of a reading. In the future, question writers will be assigned particular learning objectives on which to write questions. This encourages question writers to focus on the appropriate issues and to potentially integrate the multiple readings that support an individual learning objective.

Another outcome of the revised question-writing process is to move away from questions of the form "According to …" or "Based on …." While this form of question can help to focus candidates' answers, it doesn't necessarily fit the strategy of testing learning objectives rather than syllabus readings. It can also narrow the acceptable range of answers inappropriately. This will make grading more difficult as we begin to accept a wider range of answers, but is a more appropriate way to test whether candidates have met the required learning objectives.

Pass Mark Panels
The goal of pass mark selection is to identify the dividing line between candidates who have met the required learning standard and those who have not. Ideally, this selection process should be consistent over time (assuming no change in the required standard) and should identify the same group of successful candidates regardless of the difficulty of a particular exam. In reality there is no bright line that identifies successful candidates and ensures consistency over time.

In determining pass marks, the CAS has traditionally relied on a trend-analysis process that compares the performance on the current year exam with prior exams and attempts to determine how much current performance was driven by the exam difficulty (which should not affect the passing percentage) versus the candidate's preparedness (which should affect the passing percentage). This method works reasonably well for limited periods of time, where significant changes in candidate preparedness are unlikely and there's little chance of significant cumulative rounding error in the trend analysis. However, over longer periods of time, it's important to renormalize the required performance standard, and Chauncey's consultants noted that the CAS process could be improved in this respect.

Within the field of professional testing, the standard method for selecting pass marks is to convene a panel of experts not involved in constructing the exam and have them determine how successful versus unsuccessful candidates would be expected to perform on the exam without consideration of performance on past exams. This process requires defining a "minimally qualified candidate" (i.e., a candidate with just enough capability to merit passing the exam) and the expected capabilities of such a candidate in view of the required learning objectives for a particular exam.

Once the panel has agreed on the definition of the minimally qualified candidate, each panelist individually reviews the exam and estimates question by question how they would expect a group of 100 minimally qualified candidates to perform (i.e., how many of the 100 would answer a multiple-choice question correctly or how the 100 candidates would distribute across the range of potential scores on an essay question). The distribution of the judges' ratings for each question is then shared with the group and the rationale for outlier estimates is discussed. Following the discussion, each panelist has the opportunity to adjust their initial rating of each question. The cumulative average of the judges' ratings over all questions then provides an estimate of the appropriate passing standard on the exam.

This alternative has its own weaknesses. When reviewing individual exam questions, it is difficult to take into account the impact that candidate fatigue and exam length will have on actual candidate performance. And there is a natural tendency for panelists to tend toward a "should" orientation (in an ideal world, all successful candidates should get all questions correct, assuming the questions test the relevant learning objectives) rather than the desired "will" orientation (in reality, successful candidates will make careless errors, misread questions, forget things that they know how to do, and the like). Throughout the process, the judges are frequently reminded to estimate performance under exam conditions, but this can be difficult to assess.

The CAS piloted a panel review process for the Fall 2001 Exams and has continued to use this process for all nonjoint exams since then (the joint exams have used a similar process since the new exams were introduced in spring 2000). The traditional method has also been continued, giving exam officers two independent estimates to use in making the final pass mark determination. With a couple exceptions, the committee has not found significant differences between the two estimates, although differences of a point or half-point can be very important for individual candidates. As we gain more experience with the panel method, we'll determine if this should become a permanent part of the process or be used more for periodic renormalization.

Other Improvements
The Chauncey Project has led to a spirit of innovation that has generated other improvements. The Exam Committees have been more diligent (with mixed success) in trying to develop exams that are not too long for the allotted time. In addition, the committees have begun to experiment with parts of exams being open-book. For instance, the auto policy form in the Exam 5 reading materials and the rating manuals in the Exam 9 reading materials are now provided with the exams so that questions will focus more on real-life applications and less on memorization of particular details.

Overall, the Admissions Committees are optimistic that the Chauncey Project and other initiatives will lead to significant improvements in the admissions process and will have long-term benefits for the Casualty Actuarial Society.

Click here to write a Letter to the Editors