Examining the Process—Part II
Grading Timelineby Arlene F. Woodruff, FCAS
"Time has now expired, put your pens or pencils down. No further writing is allowed." For CAS candidates, the last exam is over and the waiting begins. For members of the CAS Examination Committee, the waiting is over and the second phase of work begins.
Week 1—Party Time! Put away the books! I nailed it!
At the CAS office, the packages from the exam centers arrive and well over 3,000 exams are logged in. Signatures must be verified and the candidate numbers are checked against the proctor’s report. As each envelope is opened the candidate’s number must be checked against the number on the short answer card (both the written number and the coded number) and on each of the essay sheets. The short answer cards are prepared for scanning and the essay sheets for all the candidates must be sorted so that individual questions can be photocopied for the graders.
Week 2—Catch up on laundry, cleaning, sleeping, and work. I did very well on the exam.
Essay questions are sent to a printer for photocopying. While these are being copied the short answer cards are checked manually. The CAS office staff will also randomly check each series of cards to make sure the scanner is working properly. When the essays return from the printer, the copies are packaged and sent to the individual graders. Original sheets are sent to the exam part chair. Any comments on ambiguous or defective questions are forwarded to the graders, exam part chair, and the vice chair of the exam series for review.
Weeks 3 and 4—Normal life at last. I think I did well on the exam.
Each essay question on the exam is sent to two graders. Each grader is given two, three or four questions to grade. Since there can be 300 to 800 answer sheets for each question, a grader will be looking at a huge pile of papers to evaluate. A suggested answer key exists for each question, but alternative solutions may be correct, and the grader must be open to different approaches to a problem. If any candidate has complained about a question, the graders and the part chair will have to consider the validity of the complaint. About two dozen responses are graded and then the results are compared. The grading partners will establish a consistent grading scale and then evaluate the solution key. Consistency and accuracy are the most important factors in grading the questions. After looking at hundreds of papers, it is possible that a grader could slightly shift focus (either harder or easier). To minimize the chance of this happening the graders will begin grading at different points on the candidate list, then when the two grades are compared any significant differences will be checked. Each grader prepares a diskette with each candidate’s number and the score for each question.
Week 5—Time to start gathering material for the next exam. I hope I did well.
The core of the grading process is the grading session. The first step is running the data through a standard grading program, verifying the data, and noting any significant discrepancies. For each candidate and each question the scores of each grading partner must be within a prescribed tolerance. If the scores do not fall within this tolerance the partners must discuss the candidate’s answer sheet and come up with a decision on what the point value should be. When all the questions have been reconciled to the required tolerance the scores are totaled and a tentative pass score is selected based on various statistics and guidelines.
This triggers the second round of reconciliation. Any candidates who have scores within a certain number of points from the tentative pass score will have all of their answers reconciled completely. This gives an exact score for any candidate near the passing score. The scores for any candidates who are close to passing will be checked manually as well. The committee will then look at the statistics one more time and make a final recommendation for the passing score.
Weeks 6 and 7—Results will be coming out in a couple of weeks. What if I blew it?
After the grading session, the part chair will submit a report to the vice chair of the exam series and the chair of the Examination Committee. In the report he will recommend a passing score, give a detailed analysis of the exam, and note any unusual questions or situations which required special handling. The committee chair and vice chair will set up a teleconference with the Vice President–Admissions to discuss all of the exams from the series and finalize the passing scores.
Week 8—Are the results out yet? I probably blew it!
After the passing score has been approved by the Vice President - Admissions, the data are released to the CAS office to prepare the mailings and pass lists.
Week 9—Is my number on the list? YES!!!