Precision Control: How Many Samples Are Enough

The business question

A small manufacturing project runs for nine years (2027–2035): sales start at an uncertain volume and grow at an uncertain rate, each unit sells at an uncertain price and costs an uncertain amount, and fixed costs are uncertain too. The NPV at 15% is simulated.

Before reporting it, the analyst sets precision requirements, for example 99% confident that the mean is known to within $5 million, and checks them with ModelRisk's precision functions, which also say how many more samples each still needs. With precision control switched on, VoseSimStop ends the simulation as soon as every requirement is met.

Who it is for

Anyone who runs simulations and must say how precise the results are, or choose the number of samples.

Main outputs

The project's NPV: mean, spread, percentiles and the chance of a loss · for four precision requirements: met or not, the precision achieved, the samples needed (by formula) and ModelRisk's estimate of the extra samples · the net cash flow by year.

How the model works

Four sheets after this one, read from left to right. Each one feeds the next.

1. 1

   Inputs sheet

   Triangular ranges for variable cost, price, first-year volume, growth and fixed costs; the tax and discount rates; the strength of the link between the two costs; four precision requirements.

2. 2

   Cash flow sheet

   Each year: volume × (price − variable cost) less fixed costs, after tax. The NPV discounts years 2028 on at 15%. Variable and fixed costs come from a rotated Frank copula: when one is high, the other tends to be low.

3. 3

   Precision sheet

   Each requirement as a VosePrecision function over two cells: TRUE or FALSE, and the extra samples it needs. VoseSimStop turns each into a stop condition, active when the switch on the Inputs sheet is 1.

4. 4

   Results sheet

   The NPV's statistics; for each requirement the precision achieved and the samples needed, by formula and as ModelRisk estimates them.

ModelRisk techniques

• Precision targets VosePrecisionMean
• Automatic stop VoseSimStop
• Standard error VoseSimMSE
• Rotated Frank copula VoseCopulaBiFrank

Find similar models

Product

• ModelRisk

Type

• Simulation

Application

• Statistical techniques

Uses

• Triangle
• Copulas
• Precision control

Inputs and assumptions

Every assumption in the model, with its uncertainty and where it comes from. The figures are illustrative.

Units and conventions

$ thousand per unit, $ million

Price and variable cost per unit in $ thousand; fixed costs, cash flows and the NPV in $ million.

Precision ± p at confidence c

The true value (with endlessly many samples) lies within ± p of the simulated value with probability c.

Key assumptions and limitations

Cash flow

Net cash flow = (volume × (price − variable cost) ÷ 1,000 − fixed costs) × (1 − tax). The original's formula floored it at zero but compared amounts in different units, so the floor never applied: a loss year counts in full here too, as in the original's results.

Years

2027–2035 (the original ran 2012–2020).

Precision formulas

The samples needed by formula use the normal approximation for the mean and the probability, the kurtosis for the standard deviation, and the density at the percentile.

Data

The ranges, the copula (Frank 20, rotated) and the requirements are those of ModelRisk's earlier Precision Control example.

ModelRisk build

In the build this example was made with, a simulation ended early by VoseSimStop leaves VoseSimPercentile and VoseSimProbability without results ('No simulation results'), although the samples are saved. The stop conditions are therefore behind a switch, off by default.

How to use it

1. Set the requirements on the Inputs sheet: a precision (in the NPV's units, or a probability) and a confidence.
2. Run a simulation (10,000 samples with a fixed seed are set) and read which requirements are met. To stop automatically, switch precision control on (P% on the ModelRisk ribbon), set a large number of samples and start: the simulation ends when every VoseSimStop condition is met.
3. Report each result with its precision, for example: mean NPV 170 ± 3 ($ million, 99% confidence).

ModelRisk functions used

VoseTriangle

The five uncertain quantities, two of them through the copula's U.

VoseCopulaBiFrank

Two linked probabilities; direction 2 rotates the copula, so the link is opposite.

VosePrecisionMean, VosePrecisionStdev, VosePrecisionPercentile, VosePrecisionProbability

Whether a statistic is known to a precision with a confidence; over two cells, the extra samples needed.

VoseSimStop

Ends a simulation once all its conditions are met (precision control switched on).

VoseSimMSE

The standard error of the simulated mean.

VoseSimMean, VoseSimStdev, VoseSimKurtosis, VoseSimPercentile, VoseSimProbability

Statistics of the simulation.

Further reading

• How many Monte Carlo iterations do you need? (Vose guide)
• Vose, D. (2008). Risk Analysis: A Quantitative Guide, 3rd edition. Wiley: how many iterations to run.