Incorporating differences in expert opinions

See also: Modeling expert opinion introduction, Distributions used in modeling expert opinion, Vose Combined Distribution

Experts will sometimes produce profoundly different probability distribution estimates of a parameter. This is usually because the experts have estimated different things, made differing assumptions or have different sets of information on which to base their opinion. However, occasionally two or more experts simply genuinely disagree. How should the analyst approach the problem?

The difference in opinion is another source of uncertainty, so should not be discounted by for example, taking the average of the opinions, or the largest (or smallest).

Instead, one needs to create a composite distribution that reflects the range and emphasis of each opinion and our confidence in the estimators.

The technique behind this calculation is to use a  Discrete({xi},{pi}) distribution where the {xi} are the expert opinions and the {pi} are the weights given to each opinion according to the emphasis one wishes to place on them.

This technique is implemented in ModelRisk with the VoseCombined({Distributions}, {Weights}) function and related probability calculation functions.

The figure below illustrates an example combining three differing opinions but where expert A is given twice the emphasis of the others due to her greater experience:

Incorrect approaches
  • Pick the most pessimistic estimate. This is generally unsatisfactory as a risk analysis model should be attempting to produce an unbiased estimate of the uncertainty. The caution should only be applied at the decision-making stage after reviewing the risk analysis results.

  • Take the average of the two distributions. This is incorrect as the resultant distribution will be too narrow. By way of illustration, consider the test situation where both experts believed a parameter should be modelled by a Normal(100, 10) distribution. Whatever technique was used to combine their opinions, the result should be the same Normal(100, 10) distribution. The average of these two distributions, i.e. AVERAGE(Normal(100, 10), Normal(100, 10)), would be a  Normal(100,10/SQRT(2)=Normal(100,7.07) from the Central Limit theorem. In other words, we would have produced far too small a spread.

  • Multiply together the probability densities at each x-value. This is incorrect because (a) it produces combined distributions with exaggerated peakedness, (b) the area under the curve is no longer 1 and (c) the combined distribution is contained between the highest minimum and the lowest maximum.

Example model

In the following model, four expert estimates are combined to construct the one estimate. The advantage of this function is it then allows one to perform a sensitivity analysis on the estimate as a whole: if you were to use the Discrete({Distributions}, {Weights}) method ModelRisk would, in this case, be performing a sensitivity analysis of five distributions: the four estimates and the Discrete distribution, which will dilute the perceived influence of the combined uncertainty.

Example model Combining_expert_opinions  Combining weighted SME estimates using VoseCombined functions.

In the above model, the VoseCombined function generates random values from a distribution constructed by weighting the four SME estimates. The weights do not need to sum to one: they will be normalised. The VoseCombinedProb function calculates the probability that this distribution assigns to the variable being less than 14. Note that the names of the experts is an optional parameter: this simply records who said what and has no affect on the calculation, but select Cell E8 and then click the Vf (View Function) icon from the ModelRisk toolbar the Vose Combined Distribution window opens, which allows us to compare each SME's estimate and see how they are weighted.

Read on:  A subjective estimate of a discrete quantity



Monte Carlo simulation in Excel. Learn more


Adding risk and uncertainty to your project schedule. Learn more



For Microsoft Excel

Download your free copy of ModelRisk Basic today. Professional quality risk modeling software and no catches

Download ModelRisk Basic now


For Primavera & Microsoft Project

Download your free copy of Tamara Basic today. Professional quality project risk software and no catches.

Download Tamara Basic now