Comparison of four methods for aggregating judgments from multiple experts
This report describes a study that compares four different methods for aggregating expert judgment data given from multiple experts. These experts need not be a random sample of available experts. The experts estimate the same unknown parameter value. Their estimates need not be a representative set of sample values from an underlying distribution whose mean is an unknown parameter, {theta}. However, it is desired to combine the experts' estimates into a single aggregation estimate to reflect their amount of available knowledge about the unknown parameter. Many different aggregation estimators and methods have been proposed in the literature. However, few have been used, tested, or compared. Four different methods are chosen for this study which have been used or proposed for use in NRC studies. The set represents a cross section of the various types of methods. The results of this study do not indicate the use of any one method over another. Methods requiring minimal decision maker input are sensitive to the biases in the experts' responses. For these methods, there is no mechanism to adjust the experts' estimates to account for any known biases in the expert population such as optimism or pessimism. The results of this study indicate that these methods tend to perform poorly in all but the most ideal cases. Conversely, methods requiring extensive decision maker inputs are sensitive to misspecification. These methods perform poorly unless complete information is known about all the experts. That is, the decision maker's input parameters must nearly equal the actual values.
- Research Organization:
- Los Alamos National Laboratory (LANL), Los Alamos, NM (United States)
- Sponsoring Organization:
- USNRC; Nuclear Regulatory Commission, Washington, DC (USA)
- DOE Contract Number:
- W-7405-ENG-36
- OSTI ID:
- 5790995
- Report Number(s):
- LA-UR-91-1512; CONF-9108100-2; ON: DE91013522
- Resource Relation:
- Conference: American Statistical Association (ASA) annual meeting, Atlanta, GA (USA), 18-22 Aug 1991
- Country of Publication:
- United States
- Language:
- English
Similar Records
On the use of expert judgments to estimate the pressure increment in the Sequoyah containment at vessel breach
To drill or not to drill: a synthesis of experts' judgments