Bayesian Model Averaging in Excel | Vose Software

ModelRisk example · General techniques

Bayesian Model Averaging

When the data cannot tell which model is right, keep them all, weighted by how well each fits.

  • Complexity: ★★★★★
  • Free Excel workbook (GU-20)
  • Needs ModelRisk: free trial

From the model’s saved run

Distribution: Normal or Lognormal?

The BMA distribution against each candidate alone (simulated).

ModelWeightMean5th pct95th pctBelow 0
Normal fit29.9%2.320.644.011.1%
Lognormal fit70.1%2.330.974.490.0%
BMA distribution2.300.874.270.5%
BMA density at x / cumulative probability at x0.37100.5974
Normal, Lognormal and their average

The business question

Fitting software ranks candidate models and invites us to pick the best. With few data the ranking is fragile: the second-best model may be nearly as plausible. Bayesian model averaging (BMA) keeps every candidate and weights it by Bayes' theorem: a prior weight (our belief before seeing the data) times the likelihood of the data under the fitted model.

Each simulation then draws from one candidate, chosen with those weights. The model does this three times, on three data sets: two distributions fitted to 11 values, five copulas fitted to 23 joint observations of three variables, and three time series fitted to 44 periods.

Who it is for
Anyone fitting distributions, correlations or time series to limited data, where the choice of model matters more than the fit statistics can settle.
Main outputs
For a distribution, a copula and a time series: each candidate's log-likelihood and its weight in the average · the averaged (BMA) distribution next to its candidates, with the density and cumulative probability at chosen values · the BMA copula's rank correlations against the data's · the BMA forecast, 25 periods ahead.

How the model works

Five sheets after this one, read from left to right. Each one feeds the next.

  1. Inputs sheet

    A prior weight for every candidate model: equal weights say we had no preference before seeing the data.

  2. Distribution sheet

    Normal and Lognormal fitted to 11 values. Each fit's log-likelihood (VoseLLH) gives its weight: prior × likelihood, scaled to add up to 100%. VoseBMAProb gives the average's density and cumulative probability (a check confirms they equal the weighted candidates'). Each sample comes from a candidate drawn with its weight.

  3. Copula sheet

    Frank, Gumbel, Clayton, Normal and Student t copulas fitted to 23 joint observations of three variables, weighted the same way. Each sample comes from a copula drawn with its weight.

  4. Time series sheet

    AR1, GBM and MA1 fitted to 44 periods; each simulation forecasts 25 periods ahead with a candidate drawn with its weight.

  5. Results sheet

    The weights, the BMA distribution against its candidates, the copula's rank correlations against the data's, and the BMA forecast.

ModelRisk techniques

  • Fit objects VoseNormalFitObject
  • Log-likelihood VoseLLH
  • Model averaging VoseBMA
  • Copula averaging VoseCopulaBMA
  • Time-series averaging VoseTimeBMA
  • Prior weights
  • Drawing a candidate VoseDiscrete

Find similar models

Product

The results in detail

Copula: which dependence?

Each fitted copula's weight, and the rank correlations it produces (simulated), against the data's.

CopulaLog-likelihoodWeightρ 1–2ρ 1–3ρ 2–3
Data0.770.720.44
Frank13.330.0%0.680.700.69
Gumbel17.802.3%0.690.700.69
Clayton18.534.7%0.690.690.70
Normal19.9639.1%0.810.780.56
Student t19.8753.9%0.790.760.55
BMA copula0.790.760.56

Time series: which model?

The weights of the three fitted time series.

BMA forecast — 10th / 50th / 90th percentile band
Time seriesPrior weightLog-likelihoodWeight
Autoregressive, order 1 (AR1)133.0534.7%
Geometric Brownian motion (GBM)232.1829.1%
Moving average, order 1 (MA1)133.0936.2%

How to read these results

  • Distribution: the 11 values are 2.34 times as likely under the fitted Lognormal as under the Normal, so with equal priors BMA gives the Lognormal 70% of the weight and the Normal 30%. The average keeps some of the Normal's left tail: a chance of about 0.4% of a negative value, against about 1.3% for the Normal alone and none for the Lognormal.
  • Copula: the Normal and Student t copulas fit best and take 93% of the weight. The Student t has the larger share only because of its higher prior (3 against 2). Only they can give each pair its own correlation. Frank, Gumbel and Clayton have one parameter for all three pairs and give every pair about 0.69, while the data have 0.77, 0.72 and 0.44.

    The BMA copula follows the data's pattern.

  • The fitted Clayton parameter is negative. In ModelRisk that is the Clayton copula turned around: the dependence is in the upper tail (large values together), which suits these data, whose largest values come together.
  • Time series: the three models fit the 43 log returns almost equally well. AR1 and MA1 have a slightly higher likelihood, GBM the higher prior, and the weights end up 35%, 29% and 36%. Their forecasts hardly differ: the series behaves like a random walk, and the band widens with the square root of time.
  • BMA weights each model by prior × likelihood, with no penalty for extra parameters. With SIC, GBM would take most of the weight. With few data, the priors matter.
  • ModelRisk's own samplers (VoseBMA, VoseCopulaBMA, VoseTimeBMA) are shown on the calculation sheets but not used: in this build they return errors during a simulation. The model draws a candidate with its weight and samples it, which is what BMA sampling means.

See it move. Open the workbook, change an input on the Inputs sheet and run the simulation: every table and chart on this page updates in Excel.

Inputs and assumptions

Every assumption in the model, with its uncertainty and where it comes from. The figures are illustrative.

Prior weights: distributions 2 inputs

Equal weights: no preference before seeing the data.

AssumptionUnitValue
Normal distributionweight1
Lognormal distributionweight1
Prior weights: copulas 5 inputs

The original example's weights: more belief in the elliptical copulas.

AssumptionUnitValue
Frank copulaweight1
Gumbel copulaweight1
Clayton copulaweight1
Normal copulaweight2
Student t copulaweight3
Prior weights: time series 3 inputs

GBM, the plainest model, given twice the weight of the other two.

AssumptionUnitValue
Autoregressive, order 1 (AR1)weight1
Geometric Brownian motion (GBM)weight2
Moving average, order 1 (MA1)weight1
Where to read the BMA distribution 2 inputs

Values at which the density and the cumulative probability are reported.

AssumptionUnitValue
x for the densityThe original example's valuevalue2.30
x for the cumulative probabilityThe original's label said 2.4 and its formula 2.3: 2.4 is usedvalue2.40

Units and conventions

Weight
Prior weight × likelihood of the data under the fitted model, scaled so the candidates add up to 100%.
Log-likelihood
VoseLLH: the natural log of the joint likelihood at the fitted parameters; higher fits better.
x for the density / cumulative
Where the BMA distribution's density and cumulative probability are read.

Key assumptions and limitations

Candidates
The average is only over the candidates offered: a model left out gets no weight.
Weights
By the likelihood alone, as ModelRisk's BMA does: no penalty for extra parameters (SIC would add one).
ModelRisk build
In the build this example was made with, VoseBMA, VoseCopulaBMA and VoseTimeBMA return errors in every iteration of a simulation, and VoseSimulate on a VoseBMAObject repeats one value. The model therefore samples the average itself: it draws a candidate with its weight (VoseDiscrete) and samples that candidate. ModelRisk's BMA functions and objects are shown beside it.
Data
The data and prior weights are those of ModelRisk's earlier BMA example. Its labels read the cumulative probability at 2.4 and its formulas at 2.3: 2.4 is used here, as an input.

How to use it

  1. Replace the data on the three calculation sheets with your own (keep each block's size, or extend its defined name), and set the prior weights on the Inputs sheet.
  2. Run a simulation from the ModelRisk ribbon (10,000 samples with a fixed seed are set).
  3. Use the BMA object (VoseBMAObject, VoseCopulaBMAObject, VoseTimeBMAObject) wherever a single fitted model would go.

ModelRisk functions used

VoseNormalFitObject, VoseLognormalFitObject
Distributions fitted to the data, as objects.
VoseCopulaMulti…FitObject
Frank, Gumbel, Clayton, Normal and T copulas fitted to the data.
VoseTimeAR1FitObject, VoseTimeGBMFitObject, VoseTimeMA1FitObject
Time series fitted to the data.
VoseLLH
The log-likelihood of a fitted model.
VoseBMA, VoseBMAObject, VoseBMAProb, VoseBMAProb10
Sample, object, density / cumulative probability and their log10 for the BMA distribution.
VoseCopulaBMA, VoseCopulaBMAObject, VoseCopulaSimulate
The BMA copula.
VoseTimeBMA, VoseTimeBMAObject, VoseTimeSimulate
The BMA time series.
VoseSimulate, VoseProb
A sample, and a density or cumulative probability, from any distribution object.
VoseSimCorrelation
The rank correlation between two outputs in the simulation.

Further reading

  • Monte Carlo simulation explained (Vose guide)
  • Hoeting, J. A., Madigan, D., Raftery, A. E. and Volinsky, C. T. (1999). Bayesian model averaging: a tutorial. Statistical Science 14(4), 382–417.
  • Vose, D. (2008). Risk Analysis: A Quantitative Guide, 3rd edition. Wiley: fitting distributions, copulas and time series to data.