Industry: Healthcare and Epidemiology Product: ModelRisk Application: Patient Satisfaction — HCAHPS Top-Box Probability and Intervention ROI
CMS pays a value-based-purchasing bonus to hospitals whose HCAHPS overall hospital rating top-box (the proportion of patients rating their stay 9 or 10 on a 0–10 scale) clears the 78% achievement threshold — roughly $1.45M/year for a five-hospital system, with a further $2.25M tier at the 82% benchmark. The system's deterministic dashboard reports a current top-box rate of 76.4% and concludes the system is "close to the threshold." The Monte Carlo simulation on the same survey data says the probability of clearing the 78% bar in the next reporting period is roughly 22% — about one chance in five — and that the closeness of the mean is hiding a posterior wide enough that the bonus payout is mostly against. The deterministic number is right on the central tendency and wrong on the only question the CFO actually asks: should we plan on the bonus or not?
The simulated top-box distribution below is the whole argument in one chart: the 78% CMS threshold sits up in the right tail, above the mean and the median, so the bonus is achievable but not on plan.
A five-hospital regional system rebuilt its HCAHPS analytics in ModelRisk. The deterministic dashboard still drives daily ops, but the quarterly bonus-forecast cycle is now anchored on this simulated top-box distribution.
HCAHPS reports six operational sub-domains plus an overall rating. Each sub-domain top-box rate is a Beta posterior on a survey-count sample, parameterized by the actual number of "top-box" responses and "non-top-box" responses received in the rolling 12-month window:
Beta is the natural conjugate for a binomial top-box count and bounded on [0, 1] — the only defensible distribution family for a proportion. The deterministic dashboard reports just the means; the posterior widths are real and material, especially for Staff responsiveness (95% CI ≈ [57%, 70%] once the period-to-period variability the dashboard pools away is restored).
A four-year regression on the system's own data links the six sub-domains to the overall top-box rate via a logistic regression. The fitted coefficients on the logit scale rank the sub-domains by their per-percentage-point influence on the overall:
That ranking is itself an actionable finding: staff responsiveness is operationally the lowest-scoring domain and the highest-influence one. Improvement spending should follow.
The mean simulated overall top-box rate is 76.3% (median 76.4%). The P10 is roughly 73.5%; the P90 is roughly 79.0%. The CMS achievement threshold of 78% sits between the median and the P90 — meaning the bonus is achievable, but not on plan. The probability of achieving the bonus is ~22%, not the 50% a point-estimate at 76.4% would imply, because the threshold sits noticeably above the mean. The $2.25M benchmark tier at 82% is, on the current operating point, effectively out of reach (P ≈ 0%).
The tornado combines the logistic-link coefficient with the realistic move-range for each sub-domain. Staff responsiveness is the dominant lever — moving its top-box from 60% to 72% (a feasible 12-percentage-point lift via a call-light response intervention) shifts the overall top-box rate by roughly ±2.6 percentage points. Nurse communication is second; cleanliness and doctor communication come next. Discharge information is the smallest mover — the regression simply does not weight it heavily once its top-box is already in the high-80s.
The operations team scoped an $850,000/year call-light response intervention: dedicated patient-care-tech coverage in the highest-volume four units across the system, with a target staff-responsiveness top-box lift from 63.6% to ~71% over a 12-month rollout.
The simulation shows the mean overall top-box rate moving from 76.4% to 79.4%, and the probability of clearing the 78% CMS threshold moving from 22% to ~78%. The expected annual CMS bonus payout rises from roughly $322,000 (the bonus times P(achieve)) to roughly $1.20M. But once the incremental bonus is netted against the $850K intervention cost, the expected annual ROI is only ~+$25,000, with a P(ROI > 0) of ~57% — the simulation reveals that the bonus alone barely covers the spend. The funding case rests on the multi-year option value of holding the threshold once cleared, not on a first-year payback.
A bundled $1.25M alternative adds a $400K facility-cleanliness lift (overnight environmental-services coverage + supply restock) on top of the responsiveness intervention. It pushes mean top-box rate to ~80.0% and probability-of-achievement to ~86%, but only marginally improves the probability of clearing the 82% benchmark tier (from ~7% to ~13%). The marginal $400K spend on cleanliness buys threshold-achievement confidence but barely touches the benchmark tier — the simulation is what made that distinction visible to the COO.
A 76.4% point estimate hides everything an executive wants to know about a 78% bonus threshold. Monte Carlo simulation in ModelRisk turns "we're close" into a probability, a probability into a dollar value, and a dollar value into a funding decision the finance team will sign.