Industry: Pharmaceutical Product: ModelRisk Application: Drug efficacy analysis
Eight randomised controlled trials of a chronic-pain agent reported standardised mean differences ranging from 0.12 to 0.55. The naive pooled mean is 0.34, the fixed-effect summary 0.35 with a tight ±0.03 CI — clean enough for a label claim. But the I² statistic from those same eight studies is 32%, meaning roughly a third of the observed between-study variance is real heterogeneity rather than sampling noise. The honest pooled answer stays near 0.34, but with a much wider predictive interval: the next study's true effect could plausibly fall anywhere between 0.19 and 0.50.
That predictive distribution — not the pooled point estimate — is the statistic regulators ask for and standard meta-analysis tools rarely produce. A clinical pharmacology team preparing a regulatory submission rebuilt their meta-analysis in ModelRisk to expose what the fixed-effect summary was hiding and to answer the operational question directly: if we run one more trial, what is the probability of a clinically meaningful readout?
The pooled posterior (orange) is the standard meta-analysis output — narrow and centred at 0.34. The predictive distribution (blue histogram) is wider, because the next study could land in a higher- or lower-responder sub-population the eight existing trials don't fully capture. The probability that the next study's true effect exceeds the SMD = 0.20 clinical-meaningfulness threshold is 96% — high enough to justify the protocol, but the spread, not the point estimate, is what the go/no-go logic depends on.
The fixed-effect inverse-variance summary assumes there is one true effect and every study estimates it with sampling noise:
\[ \hat\mu_F \;=\; \frac{\sum w_i y_i}{\sum w_i}, \quad w_i = 1/v_i, \quad \mathrm{SE}(\hat\mu_F) = \sqrt{1/\sum w_i} \]
When the studies actually disagree more than their within-study standard errors predict — different populations, durations, placebo response rates, geographies — the model is wrong and the CI is too narrow. The DerSimonian–Laird estimator catches this by adding a between-study variance τ² to each weight:
\[ \tau^2 \;=\; \max!\left(0,\; \frac{Q - (k-1)}{c}\right), \quad w_i^* = 1/(v_i + \tau^2) \]
For these eight trials, Q = 10.2 on 7 df and the resulting τ ≈ 0.07 — a real spread in true effects between studies, modest in size but enough to widen the honest interval well beyond the fixed-effect CI.
The fixed-effect diamond sits at 0.35 with a narrow ±0.03 CI; the random-effects diamond sits at 0.34 with a CI nearly two-and-a-half times as wide (±0.08). Both diamonds are pooled summaries — neither answers the question that matters at submission, which is what the next pivotal trial is likely to find.
The pooled posterior on μ tells you what we believe the mean true effect across studies is. The predictive distribution adds the between-study heterogeneity τ to that uncertainty:
\[ \theta_\text{next} \;\sim\; \mathcal{N}(\hat\mu_R,\; \tau^2 + \mathrm{SE}(\hat\mu_R)^2) \]
The 95% predictive interval is approximately [0.19, 0.50] — a range whose lower edge sits right at the SMD = 0.20 threshold the team had committed to as their go/no-go for the next protocol. That is the operationally relevant fact: the pooled CI [0.26, 0.43] makes a positive readout look near-certain, but once between-study heterogeneity is folded in, a meaningful slice of the predictive mass sits below the bar.
The probability that the next study's true effect exceeds the SMD = 0.20 threshold is 96% — high enough to justify the protocol, but not the near-certainty the pooled CI implies. The fixed-effect view, which ignores τ entirely, put that probability at essentially 100% and would have shaped the next trial's sample size and risk assumptions too optimistically.
Plausible moderators that could explain why effects vary between studies:
Baseline severity is the dominant suspect — three of the larger-effect trials enrolled more severe patients, where a regression-to-the-mean component adds to the apparent effect. Trial duration is second (longer trials show greater placebo drift, narrowing the difference). Endpoint definition is small but real: patient-reported outcomes consistently show larger effects than clinician-scored ones in this indication.
These moderators are candidates for a meta-regression that the team did not have power to run with eight studies. The simulation, however, can stress-test what happens to the predictive distribution if τ is 50% larger — the predictive interval widens to [0.13, 0.56], and P(next true effect > 0.20) drops to 91%. That sensitivity is what the protocol's risk assessment should reflect.
The fixed-effect CDF is steep and centred — a confident answer that ignores the heterogeneity actually present in the data. The random-effects posterior on μ is wider, but still centred and narrow at the tails. The predictive distribution is the only one that can answer the operational question — "if we run one more trial, what is the probability of a positive readout?" — and it is the answer the protocol's go/no-go logic depends on.
The pooled mean answers the wrong question — "what is the average across what we've already done?" — when the operational question is "what will the next trial find?" Random-effects meta-analysis in ModelRisk produces both numbers, and forces the discussion that should happen at every gate: how much of our remaining uncertainty is about μ, and how much is about τ?