 |
Download a pdf copy of this help file here |
|
Download a pdf copy of this help file here |
See also: Modeling expert opinion introduction, Distributions used in modeling expert opinion, Vose Combined Distribution
Experts will sometimes produce profoundly different probability distribution estimates of a parameter. This is usually because the experts have estimated different things, made differing assumptions or have different sets of information on which to base their opinion. However, occasionally two or more experts simply genuinely disagree. How should the analyst approach the problem?
The difference in opinion is another source of uncertainty, so should not be discounted by for example, taking the average of the opinions, or the largest (or smallest).
Instead, one needs to create a composite distribution that reflects the range and emphasis of each opinion and our confidence in the estimators.
The technique behind this calculation is to use a Discrete({xi},{pi}) distribution where the {xi} are the expert opinions and the {pi} are the weights given to each opinion according to the emphasis one wishes to place on them.
This technique is implemented in ModelRisk with the VoseCombined({Distributions}, {Weights}) function and related probability calculation functions.
The figure below illustrates an example combining three differing opinions but where expert A is given twice the emphasis of the others due to her greater experience:
Pick the most pessimistic estimate. This is generally unsatisfactory as a risk analysis model should be attempting to produce an unbiased estimate of the uncertainty. The caution should only be applied at the decision-making stage after reviewing the risk analysis results.
Take the average of the two distributions. This is incorrect as the resultant distribution will be too narrow. By way of illustration, consider the test situation where both experts believed a parameter should be modelled by a Normal(100, 10) distribution. Whatever technique was used to combine their opinions, the result should be the same Normal(100, 10) distribution. The average of these two distributions, i.e. AVERAGE(Normal(100, 10), Normal(100, 10)), would be a Normal(100,10/SQRT(2)=Normal(100,7.07) from the Central Limit theorem. In other words, we would have produced far too small a spread.
Multiply together the probability densities at each x-value. This is incorrect because (a) it produces combined distributions with exaggerated peakedness, (b) the area under the curve is no longer 1 and (c) the combined distribution is contained between the highest minimum and the lowest maximum.
In the following model, four expert estimates are combined to construct the one estimate. The advantage of this function is it then allows one to perform a sensitivity analysis on the estimate as a whole: if you were to use the Discrete({Distributions}, {Weights}) method ModelRisk would, in this case, be performing a sensitivity analysis of five distributions: the four estimates and the Discrete distribution, which will dilute the perceived influence of the combined uncertainty.
Example model 
Combining_expert_opinions Combining
weighted SME estimates using VoseCombined functions.
In the above model, the VoseCombined function generates random values from a distribution constructed by weighting the four SME estimates. The weights do not need to sum to one: they will be normalised. The VoseCombinedProb function calculates the probability that this distribution assigns to the variable being less than 14. Note that the names of the experts is an optional parameter: this simply records who said what and has no affect on the calculation, but select Cell E8 and then click the Vf (View Function) icon from the ModelRisk toolbar the Vose Combined Distribution window opens, which allows us to compare each SME's estimate and see how they are weighted.