Industry: Telecommunications and IT Product: ModelRisk Application: Cloud Service Reliability
A "four nines" SLA — 99.99% availability — buys the customer 52.6 minutes of downtime per year. Miss it and the service-credit clauses, often $10k–$40k per minute over budget for enterprise contracts, trigger immediately. For a regional cloud operator running tens of thousands of tenants, the question is not "what is our average uptime?" — it is "what is the probability the cumulative downtime budget gets blown, and which failure mode is most likely to do it?" That is a tail question, and a deterministic MTBF/MTTR worksheet cannot answer it.
A Tier-1 cloud-services unit rebuilt its reliability view in ModelRisk: four independent failure sources — hardware faults, software incidents, DDoS events, and demand-surge saturation — composed into a single annual-downtime distribution against the 52.6-minute SLA budget. The deterministic worksheet adds the expected contributions to roughly 27 minutes — comfortably inside budget. The simulation tells a different story: the mean lands near 27 minutes, but P90 ≈ 57 min and P99 ≈ 122 min, and the probability of breaching the 52.6-minute budget is ~12% per year. A "comfortable" average conceals a roughly 1-in-8 chance of triggering service-credit clauses.
The skew comes from software-incident MTTR: the rare long production rollback is what eats the budget, not the routine hardware blips. Below, each failure source is built up separately and summed per simulated year.
Steady-state availability is the textbook ratio A = MTBF / (MTBF + MTTR), but in a distributed cloud no single MTBF/MTTR pair describes the system — different fault classes have different rates and different recovery shapes. Each is modelled separately and summed per simulated year.
A = MTBF / (MTBF + MTTR)
LogNormal is the right choice for MTTR — fixing an incident is the product of many multiplicative factors (paging delay × diagnosis time × rollback time × verification), and that compounding is exactly what produces a right-skewed distribution.
Adding the expected per-source contributions gives an expected annual downtime of roughly 27 minutes — comfortably inside the 52.6-minute budget. From that number alone, the SLA looks safe. But the distribution above shows the budget is breached ~12% of years, because the right tail of software-incident MTTR carries far more mass than the mean suggests. A deterministic MTBF/MTTR worksheet collapses that tail into a single reassuring average and hides the 1-in-8 SLA-credit exposure entirely.
Sensitivity on the P99 annual downtime tells the SRE team where the next dollar should go.
Software-incident MTTR dominates, followed by the demand-peak / capacity ratio and the DDoS event rate. Hardware fault rate — the thing classical reliability worksheets obsess over — ranks near the bottom because failover already trims it. The takeaway: a faster rollback pipeline (canary deploys, automatic rollback on SLI breach) buys more uptime per dollar than swapping out servers.
The team scoped a $10M, three-quarter programme: canary-deploy infrastructure with auto-rollback (cuts software MTTR median 8 → 5 min), a regional always-on DDoS scrubbing service (cuts cyber MTTR 12 → 5 min), and rack-level failover automation (cuts hardware MTTR 2.5 → 1.3 min).
After the programme, P(SLA breach) drops from ~12% to ~1%, mean annual downtime falls from 27 to ~14 min, and P99 from 122 to ~55 min. That converts directly into expected SLA-credit savings.
Revenue for this service runs ~$8,000/min; SLA credits add ~$40,000/min above the 52.6-minute threshold. Putting both into the simulation produces the annual-cost distribution:
The mean is a modest ~$0.35M, but VaR99 reaches $3.7M and the expected shortfall beyond it is $5.8M — the tail dominated by the rare bad-deploy plus DDoS double-hit, with a 1.6% chance of exceeding $3M in a year. That is the number the CFO needed to see to greenlight the reliability programme: the $10M spend pays for itself by clipping the right tail of this distribution, not by changing the average.
The mean uptime number is the wrong question for a four-nines SLA. The right question is the probability of breaching the 52.6-minute budget — and Monte Carlo turns that into a single, defensible percentage that maps directly to dollars at risk.