How many samples to run

Download a pdf copy of this help file  here

See also: Monte Carlo simulation introduction

A question that naturally arises when doing MC simulation, is the following. Can we determine how many samples to run a Monte Carlo model for?

It's a good question, but the answer depends on a few other things you need to work out first. In the 'old' (i.e. maybe ten years ago) computers ran much more slowly, of course. The first spreadsheet simulation models, which ran on Lotus 1-2-3, would take hours to run a few hundred samples on very simple models. Of course, there is a certain propensity to produce more detailed and computer-intensive models when you know that the computer can handle it, but for most applications the computer simulation speed is no longer always the over-riding concern any more.

In general, you'll have two opposing pressures:

•  Too few samples and you get inaccurate outputs, graphs (particularly histogram plots) that look 'scruffy';

•  Too many samples and it takes a long time to simulate, and it may take even longer to plot graphs, export and analyze data, etc afterwards. Export the data into Excel and you may also come upon row limitations, and limitations on the number of points that can plotted in a chart.

For general risk models, there is certainly a minimum number one should run. More technical arguments can be put into practice for determining how many samples you need to run for a particular level of accuracy of some reported statistic, discussed below.

There will usually be one or more statistics that you are interested in from your model outputs, so it would be quite natural to wish to have sufficient samples to ensure a certain level of accuracy. Typically, that accuracy can be described in the following way:

'I need the statistic Z to be accurate to within +/- d with confidence a.'

Further in this topic is described how you can determine the number of samples you need to run to get some specified level of accuracy for the most common statistics:

•  The mean

•  Cumulative probability

The examples contain models to let you monitor the level of accuracy in real time. Note that all models assume that you are using Monte Carlo sampling. This will therefore somewhat over-estimate the number of samples you'll need if you are using Latin Hypercube sampling.

Minimum number of samples to run

A short answer, burdened with many caveats, is to run no less than 300 samples if we are interested in the entire output distribution. At 300 samples you start to get a reasonably well defined cumulative distribution, so you can approximately read off the 50th and 85th percentiles, for example, and the mean is pretty well determined for most output distributions.

At the same time, if you export the generated values from two or more random variables in your model to produce scatter plots, 300 is really the minimum you need to get some sense of the patterns that they produce (i.e. their joint distribution).

We at Vose usually have our models set to run 3000 samples as a default (but obviously increase that figure if a particularly high level of accuracy is warranted), because we plot a great deal of scatter plots from generated data, and this is about the right number of points before the scatter plot gets clogged up, and certainly enough for all the percentiles and statistics to be well-specified.

The figures below show what type of variation you would typically get for a cumulative distribution between runs of 300 samples, and of 3000 samples. Since most models include an element of guess work in the choice of model, distributions or parameter values to use, one should not usually be too concerned about exact precision in the Monte Carlo results, but you'll see that 300 samples is probably the least level of accuracy you might find acceptable.

Now these figures below show the same input and output plotted together as a scatter plot for 300 and 3000 samples. We find scatter plots to be a great, intuitive presentation of how, among others, the input variability influences the output value. You'll see that the pattern is just about visible for 300 samples, and just starting to get clogged up at 3000 samples (of course, if you run more than 3000 samples, you can plot a sample of just 3000 of them to keep the scatter plot clear). If the pattern were simpler, the left panel of 300 samples would of course be clearer.

Getting sufficient accuracy for the mean

Monte Carlo simulation estimates the true mean m of the output distribution by summing all of the generated values xi and dividing by the number of samples n:

If Monte Carlo sampling is used, each xi is an independent sample from the same distribution. Central Limit Theorem then says that the distribution of the estimate of the true mean is (asymptotically) given by:

where s is the true standard deviation of the model's output.

Using a statistical principle called the pivotal method we can rearrange this equation to make it an equation for m:

Figure 1 shows the cumulative form of the Normal distribution for Equation (1). Specifying the level of confidence we require for our mean estimate translates into a relationship between d, s, and n as you can see from Figure 1:

Figure 1

More formally, this relationship in Equation (1) is:

where Ф-1(•)  is the inverse of the standard Normal cumulative distribution function (i.e. with mean 0 and standard deviation 1). Rearranging (2) and recognizing that we want to have at least this accuracy gives a minimum value for n:

We have one problem left: we don't know the true output standard deviation s. It turns out that we can estimate this perfectly well for our purposes by taking the standard deviation of the first few (say 50) samples.

Getting sufficient accuracy for the cumulative probability P(x) associated with a particular value x

Percentiles closer to the 50th percentile of an output distribution will reach a stable value relatively far quicker than percentiles towards the tails. On the other hand, we are often most interested in what is going on in the tails because that is where the risks and opportunities lie. For example, Basel II and credit rating agencies often require that the 99.9th percentile or greater be accurately determined. The following technique shows you how you can ensure that you have the required level of accuracy for the percentile associated with a particular value.

ModelRisk for the MR help file will estimate the cumulative percentile Px of the output distribution associated with a value x by determining what fraction of the samples fell at or below x. Imagine that x is actually the 80th percentile of the true output distribution. Then, for Monte Carlo simulation, the generated value in each sample independently has an 80% probability of falling below x: it is a binomial process with probability p = 80%. Thus, if so far we have had n samples and s have fallen at or below x, the distribution Beta(s+1, n-s+1) described the uncertainty associated with the true cumulative percentile we should associate with x.

When we are estimating the percentile close to the median of the distribution, or when we are performing a large number of samples, s and n will both be large, and we can use a Normal approximation to the Beta distribution:

Rearranging (4) and recognizing that we want to have at least this accuracy gives a minimum value for n:

By monitoring s and n we can determine whether we have reached the required level of accuracy using either Equation (3) or (4).

Read on: About random number generation seeds