Incorporating Differences
in Expert Opinion
Expert opinion is an important source of information for quantifying model parameters and variables. But often expert estimates can produce unrealistic distributions. Therefore, you need to follow a number of important principles to get the most reliable and unbiased estimate (see M0300). Incorporating differences in expert opinion in the model is one often used and valuable technique that can help you avoid errors in modeling expert opinion.

This Risk Analysis Tip will show you how to combine expert opinion the correct way using @RISK and also give you tips on how to avoid errors.

Correct way of incorporating differences in expert opinion
Experts will sometimes produce profoundly different estimates for probability distributions and parameters. This is often because experts have estimated different things, made different assumptions or have accumulated different sets of information on which to base their opinions. Occasionally two or more experts simply genuinely disagree. How should you approach this problem?

One way to approach this problem is to treat the difference in opinion as another source of uncertainty. Therefore, the differences should not be discounted by, for example, taking the average of the opinions or the largest (or smallest) opinion. Instead, you need to create a composite distribution that reflects the range and emphasis of each opinion and confidence in the estimators.

Consider an example: Imagine that you have an important uncertain value in your model and that you ask three experts to estimate it. All three experts have the same information and it has been widely disseminated but you ask each one to estimate the distribution and parameters that you need to put in your model separately. Therefore, the experts don't sit together and decide what value they think the parameter should be. Instead, they discuss the information available together, and then separately estimate it. Of the three estimates you receive, two are PERT distributions (see M0361) and one is a General distribution (see M0203) that has a customized shape. These three distributions are plotted together below:

You can see they are different and you decide to weigh expert B (brown line) twice as much as expert A (red line) or expert C (blue line). You could have given them equal weightings but this is an example to show when you have more faith in B, for example, because perhaps she is closer to the project or more experienced.

The figure below shows the results of the two ways of modeling the combined expert opinion (correct and incorrect).

Correct Way
The correct technique to use is a Discrete distribution (see M0129), where the {xi} are the expert opinions and the {pi} are the weights given to each opinion according to the emphasis one wishes to place on them.

The spreadsheet with the correct model is provided here: Combining opinions.

Incorrect Way
An often used but incorrect way of combining expert opinions is to multiply the expert opinions by their weights, sum the results and divide the outcome by the sum of the weights to normalise it.

This method is wrong is because the formula calculates a weighted average of the three opinions. Therefore, the model will always pick a value in the centre and will not give the same degree of spread that is shown with the discrete distribution. The reason why you want to use the discrete distribution is that at least one person believes that the true value should be as low as three and at least one person believes that the value should be as high as ten. As we see from the figure above (red line), the incorrect distribution does not include these values.

This ‘Risk Analysis Tip’ comes from one of the over 500 risk analysis topics available in ModelAssist for @RISK, which gives a more detailed explanation of the above methods and any risk analysis techniques involved.

Topics referenced in this tip can be found in ModelAssist for @RISK by searching the following topic codes: M0162, M0160, M011.