Industry: Telecommunications and IT
Product: ModelRisk
Application: IT System Resilience

A 4-nines payments platform that fails its budget one year in three

A payments platform with a four-nines availability target gets a downtime budget of 52.6 minutes per year. The architecture diagram — six tiers, active-passive everywhere, hot-standby database, multi-AZ — said the budget was comfortable. A chaos-engineering campaign that injected real failure modes into a staging clone said something different. When the team rebuilt the same composition probabilistically in Vose Software's ModelRisk, the 4-nines budget was breached in 30% of simulated operational years, and a P99 year saw 6.2 hours of accumulated downtime — more than seven times the budget.

The gap is not poor engineering. The gap is what happens when an architecture diagram is multiplied through in averages: redundancy, correlation, and the heavy-tailed repair-time distribution disappear, and what is left is a single optimistic number. The chart below shows it directly — the mean year sits just inside the 52.6-minute budget, but the distribution's right tail spills far past it.

Composing six tiers and one regional shock

Each of the six tiers — load balancers, auth service, payments core, database tier, message bus, CDN/edge — has its own failure intensity (events per year, Poisson) and its own repair-time distribution (LogNormal in minutes, with geometric standard deviation between 1.5 and 2.6). Redundancy is captured as a multiplicative factor on the per-tier failure rate: an active-passive pair with healthy failover cuts the effective rate to 9–11% of the bare-metal number; the CDN's anycast mesh cuts it to 7%.

Two design decisions matter at the modeling layer:

• Repair time is LogNormal, not Normal. Engineers do not get to choose a mean recovery time; most incidents close in 15–30 minutes and a stubborn few drag on for hours. The database tier's MTTR is modeled with median 34 minutes and GSD 2.6 — meaning roughly 1 incident in 150 still drags past 6 hours. A symmetric distribution buries that tail.
• A correlated regional event sits on top of the per-tier model. Cloud-region partitions, fibre cuts and provider control-plane outages take down multiple tiers at once. Modeled as Poisson(0.25 events/yr) with LogNormal severity (median 55 minutes, GSD 2.6), this single rare event is the largest contributor to mean annual downtime — invisible to any per-tier reliability calculation.

Annual downtime is the sum across all tiers and the regional shock. Availability is 1 − downtime / 525,600.

What the architecture-diagram answer missed

The deterministic resilience worksheet, built by summing each tier's mean-time-between-incidents and mean-time-to-repair, reported an expected availability of 99.992% — comfortably inside the 4-nines target with 10 minutes of headroom. The Monte Carlo run produced a mean of 51 minutes/year, a P90 of 130 minutes and a P99 of 374 minutes. The 4-nines budget just held on the mean, but breached 30% of operational years — close to one in three. The averaged-out diagram had no language for "most years fine, one year in thirty catastrophic" — yet that is exactly the operational pattern the platform was about to inherit.

Where the downtime really comes from

Ranking each source by its mean contribution to annual downtime tells the team which tier is buying them the least resilience per dollar invested.

The rare regional event is the single largest contributor — roughly 22 minutes/year on average — more than any individual tier. Among the tiers, the payments core (≈9 min/yr) and auth service (≈8 min/yr) lead, with the database tier close behind (≈7 min/yr); the payments core's MTTR is wide (GSD 2.4) because rolling back a poisoned write is sometimes a multi-hour exercise. Crucially the load-balancer (≈2 min/yr) and CDN (≈1 min/yr) tiers, where the team had been planning to invest, contribute very little. The chart redirected the next $3M of resilience capex from edge to the regional-failover and database tiers.

Active-active database and cross-region failover

Two architectural changes were modeled head-to-head against the as-is baseline:

1. Cross-region failover only. Cuts the rare regional outage rate from 0.25/yr to 0.12/yr and shortens its severity to median 48 minutes. No change to per-tier resilience. Cost: $0.7M setup, $0.2M/yr ops.
2. Active-active DB plus cross-region failover. Cuts the database tier's effective failure rate by a further ~3.6× (red factor 0.09 → 0.025) and cuts both the rate (to 0.06/yr) and the severity of the regional shock. Cost: $2.4M setup, $0.6M/yr ops.

The full package drops the mean annual downtime from 51 to 27 minutes/year, the P99 from 374 to 182 minutes, and the probability of breaching the 4-nines budget from 30% to 17%.

From minutes of downtime to dollars at risk

Downtime cost was modeled at $48,000 per minute — the industry midpoint for a mid-size payments platform once direct revenue, SLA credits, and per-incident remediation are summed. The cost distribution is more revealing than the downtime distribution, because it inherits the heavy tail and converts it into a budgeting number a CFO can act on.

Expected annual outage cost is $2.4M as-is, $1.8M with cross-region failover, and $1.3M with the full active-active package. The P95 annual cost — the figure used in the CIO's risk register — falls from $9.1M to $4.8M. The full $3.0M five-year programme pays back in expected terms inside three years and removes roughly $4M of P95 tail-year exposure.

ModelRisk functionality used

• Per-tier compound-Poisson models stacking VosePoisson event counts with VoseLognormal2 MTTR draws to build each tier's annual-downtime distribution.
• Mixture for the correlated regional event so the rare partition shock is composed on top of the per-tier model rather than averaged into it.
• Sensitivity ranking by mean downtime contribution rather than rank correlation, so that low-frequency tiers (whose per-year contribution is zero in most years) are ranked by the downtime they actually add over the long run.
• Architecture comparison via overlaid downtime distributions and cost CDFs, so the trade-off between $0.7M and $3.0M investments was read off the same picture.
• Direct minute-to-dollar conversion with a per-minute cost variable to translate engineering metrics into the CFO's risk register without rebuilding the model.

Resilience is not the diagram; it is the joint distribution of how every tier fails and how long it takes to come back, plus the rare event that takes several tiers down at once. Monte Carlo simulation in ModelRisk is what turns "we are nominally four-nines" into "we breach four-nines roughly one year in three — and here is the $3M that roughly halves that exposure."