Examining the Process—Part III
by Arlene F. Woodruff, FCAS
One of the goals of the CAS Examination Committee is to produce an exam with no "bad" questions, but, yes, some questions are faulty. And yes, the CAS Examination Committee reviews each comment on problem questions.
When CAS candidates write to the CAS Examination Committee officers about faulty questions within two weeks after the examination, the officers or committee members can thoroughly review the comments and modify the solution accordingly. Statistics are calculated on each problem to see how well the candidates answered the question. Occasionally, these can also show that a question may be faulty without a candidate having to write a comment.
The following is an actual excerpt from an Examination Committee recommendation letter pertaining to a candidate comment received after an examination. "We" refers to the examination graders.
"We received a comment on Question 5 that caused a disturbing investigation. The question gave a distribution for lx and the candidate rounded the resulting values to integers. In the Snader paper, he gives values for lx in table format and these values happen to be integers. He does not calculate the values himself. We state in our instructions that the students should keep four significant digits. In the Bowers text on page 54, it states ‘Although the values of lx have been rounded to integers, there is no compelling reason, according to (3.3.1), to do so.’ In light of this, we would argue that the candidate should not have rounded the values of lx. Now we come to the sticky part. We found a note in Parmenter in an example on page 128 stating that lx must be an integer. This obviously caused confusion with the student who wrote in. Since there is a source who advocates this method, we feel that credit should be given for both answers. This situation points out the different approaches in the two texts. We would like to discuss the possibility of removing the overlapping Parmenter sections from the Syllabus."
When the committee members find that a question has no correct solution, then the question will be thrown out. When more than one correct answer can be applied to a question, credit would be given for both. The problem occurs when different interpretations cause different answers. In these cases, the committee members must decide what the best course of action to take. They could decide that the question is so faulty that it should be thrown out. They could give credit for more than one answer. They could give credit for only the most correct answer. They could also grade the examination in multiple ways and adjust any candidates who are on the border of passing. Each situation is unique and the solutions are not uniform. Sometimes, a difference of opinion exists between what is a "reasonable" interpretation and what is a stretching point. The discussions on the best course of action are often the most time-consuming part of the grading process.
Yes, the CAS Examination Committee members continue to learn from each faulty question and try not to repeat any errors. And, finally — yes, the committee members do consider how their decisions affect the candidates.