A/B Testing & Experimentation (Healthcare)

Overview

Did the new model, workflow, or outreach actually help or did we get lucky? This skill plans and analyzes controlled experiments with the statistical rigor to answer that, including the parts teams usually skip: a pre-registered hypothesis, an honest power analysis, the right randomization unit, and guardrail metrics so a "win" on one metric isn't a loss on safety.

When to Use This Skill

• Rolling out a new predictive model, alert, UI, or care-management intervention.
• Sizing a test up front (how many members/encounters and for how long).
• Analyzing results and deciding ship / no-ship with quantified uncertainty.
• Comparing search/ranking configs (pairs with clinical-text-search-elk) or model

variants.

Method

1. Hypothesis & metrics one primary metric (the OEC), plus guardrails (e.g.,

safety, cost). Pre-register to avoid metric-shopping.

1. Randomization unit member/patient, provider, or facility; cluster-randomize

when the intervention spills across patients within a provider, and account for intra-cluster correlation.

1. Power analysis compute sample size for the minimum detectable effect at chosen

α/power before launch; avoid underpowered tests that can't conclude.

1. Analysis two-proportion / t / Mann-Whitney as appropriate; CIs not just p-values;

or a Bayesian posterior on the effect (P(better) and expected loss).

1. Sequential / peeking if monitoring continuously, use alpha-spending or always-

valid (mSPRT) methods so early stopping doesn't inflate false positives.

1. Precision & validity CUPED with pre-period covariates to cut variance; check

for sample-ratio mismatch (SRM) as a trust signal; control multiple testing (BH/FDR).

Example

import numpy as np
from statsmodels.stats.power import NormalIndPower
from statsmodels.stats.proportion import proportions_ztest, proportion_effectsize

# Sample size: baseline 12% readmit, detect a 2-point absolute drop, 80% power
eff = proportion_effectsize(0.12, 0.10)
n = NormalIndPower().solve_power(effect_size=eff, alpha=0.05, power=0.80, alternative="two-sided")
print(f"n per arm ≈ {np.ceil(n):.0f}")

# Analyze
z, p = proportions_ztest([n_ctrl_event, n_trt_event], [n_ctrl, n_trt])
print(f"z={z:.2f}  p={p:.4f}")

# Bayesian read: posterior probability treatment beats control (Beta-Binomial)
a_c, b_c = 1+ctrl_succ, 1+ctrl_fail
a_t, b_t = 1+trt_succ,  1+trt_fail
s_c = np.random.beta(a_c, b_c, 200000); s_t = np.random.beta(a_t, b_t, 200000)
print("P(treatment > control) =", (s_t > s_c).mean())

Cautions

Confounds that bite healthcare experiments: novelty/seasonality, non-compliance (analyze intention-to-treat), spillover between arms, and peeking without sequential correction. A statistically significant but clinically trivial effect is not a win pre-define a meaningful effect size. Where randomization is impossible, fall back to quasi-experimental / causal-inference designs and say so.

Outputs

• experiment_design.md hypothesis, metrics/guardrails, unit, MDE, sample size, duration.
• analysis.md effect estimate, CI, p-value/posterior, SRM check, decision.
• Power and posterior plots.

Healthcare Context

Reflects healthcare realities: clustered randomization by provider/site, regulatory and ethical constraints, intention-to-treat, and safety guardrails alongside the primary metric. Directly supports controlled, statistically valid rollout of any model built with healthcare-predictive-modeling.

References

• Kohavi, Tang & Xu (2020), Trustworthy Online Controlled Experiments.
• Deng et al. (2013), CUPED variance reduction; Johari et al. (2017), always-valid p-values.
• statsmodels power/proportion https://www.statsmodels.org