The COTOR Challenge
Committee Stimulates Thought on Risk
By Louise Francis, Chairperson, and Steven Visner, Member,
CAS Committee on Theory of Risk

In spring 2004 a member of the CAS Committee on Theory of Risk (better known as COTOR) challenged his colleagues to estimate the pure premium in the layer 500K xs 500K based on a listing of 250 claims. A number of different approaches were used and various methods for working with heavy tailed distributions were recommended. These included "Loss Models: From Data to Decisions" by Stuart Klugman, Harry Panjer, and Gordon Willmot and "An Introduction to Econophysics: Correlations and Complexity in Finance" by Rosario N. Mantegna and H. Eugene Stanley. In the end Phil Heckman emerged the winner with his recommendation to use a mixture of lognormal distributions that seemed to work even though the true distribution was log Cauchy.

While there was some concern around the selection of the sample of 250 claims, with some challenge participants feeling that some sample statistics were too far from the real distribution, COTOR feels that the solution of these types of problems are of real interest to the actuarial community. Since the data actuaries deal with is frequently heavy tailed, and the normal or lognormal assumptions that are commonly used in finance applications of similar problem does not provide a reasonable approximation to insurance distributions, COTOR believes that the frequency of extreme events is commonly underestimated and often by a large margin.

COTOR issued another challenge this past fall under well-defined conditions. Stuart Klugman picked the sample selection of 250 claims generated randomly from an inverse transformed gamma distribution. Information about the "true" distribution and its parameters was not disclosed until the presentation of solutions at the CAS Annual Meeting in November 2004. The challenge was to estimate the average severity and 95 percent confidence intervals for the $5 million xs $5 million layer.

In total eight different people responded to the challenge, submitting a total of 10 different responses. The results of this challenge were presented at the 2004 CAS Annual Meeting in Montréal. Five of the eight respondents and Phil Heckman (applying his round 1 technique to the round 2 data) presented their results and techniques to a standing-room crowd. The committee chairperson, Louise Francis, presented an award to three challenge participants, Dave Clark, Glenn Meyers, and Jonathan Evans. The awards were based on a number of factors considered together, including the accuracy and clarity of the solutions as well as the creativity used and the method's ability to lend itself to practical application.

When analyzing the submitted results for the challenge, there was a nearly 13 to 1 spread between the lowest to highest mean. All responders recognized there was tremendous uncertainty in results (range from upper to lower confidence level went from a low of eight to a high of infinity). All but two of the respondents relied on approaches commonly found in the literature on fitting distributions or modeling extreme values. Only one of the results came within 10 percent of the true mean. Interestingly enough, half the responses were below the true mean and half were above. When an obvious outlier response was eliminated and the remaining responses were averaged, the resultant average was within two percent of the true mean. The panel discussed that potential implications were for an insurance company not to rely on the results of only one model when making important decisions.

The solutions submitted warrant a few general summary comments. First a number of participants used some form of the Pareto distribution. This is not surprising, as the Pareto distribution is prominently represented in the extreme value literature. Various responders used both the single parameter Pareto, popularized by Stephen Philbrick, and a version dubbed "the Generalized Pareto" in some of the extreme value literature (there is actually more than one Generalized Pareto in the statistical literature). Many of the formulas used in the fitting of a Pareto are relatively simple to implement and the Pareto has a much heavier tail than more conventional distributions such as the lognormal. However, since the Pareto is a truncated distribution, i.e., it is fit only to data that exceed a selected threshold, there are significant issues with how to select the threshold. Different choices typically result in different parameter estimates and the different parameter estimates can result in very different estimates for excess layers of insurance.

A number of authors fit a "ground-up" distribution to the data, rather than fitting a distribution just to tail claims. In this category was the mixture approach. Mixtures of distributions are known to have heavier tails than their individual distributions have. Another approach used was to transform the data (i.e., apply a functional transform such as the log of the claims) until a distribution near to one of the more conventional densities, such as the Lognormal or Gamma, is obtained. Certain transforms, such as the inverse, log, and multiple log transforms, often result in distribu
tions with heavy tails. A third approach involved the use of kernels to approximate the distribution. The kernel approach has appeared in the statistical literature recently as a non-parametric technique for approximating densities.

Bayesian approaches were prominently featured in the responses. While frequently used to assess the variability of the expected value of the layer, Bayesian analysis was also incorporated into the mean estimate by some responders. When applying a Bayesian approach, the model parameters for claim severity are themselves assumed to be random draws from a probability distribution. Given a distribution for the parameters, either numerical integration or simulation was used to compute the severity of the $5 million xs $5 million layer. Some authors considered model risk as well as parameter risk. That is, the risk that the selected model was the wrong model was incorporated into the estimate of the layer expected value and its confidence intervals. It is not surprising that when both parameter risk and model risk are considered, some very large confidence intervals are estimated. Based on the results of the COTOR Challenge experiment, very wide confidence intervals are to be expected.

It seemed clear it is not easy to make these estimations and that real-world problems are even more challenging than this, because of trend and development (considerations that were eliminated from this challenge), potential unforeseen changes in the environment, and the fact 250 real-world claims would likely not follow any known distribution.

COTOR plans to come up with a new challenge, including some of these additional complications in the near future. More details of the challenge, including write-ups of the responses submitted, can be found online.