A/B Test Analysis

When to use this skill

The experiment has finished (or has reached planned sample size) and the user needs to interpret the results. Triggers:

• "Analyze this experiment…"
• "Did the test win?"
• "Was the lift significant?"
• "Write a readout for experiment X"
• "Compare treatment vs control on…"

If the experiment is still being planned, use ab-test-design.

Required inputs

If pre-registration is missing, flag it loudly in the readout. Post-hoc analysis without a pre-reg should be labeled exploratory.

Workflow

1. Sanity check the data.

• Verify variant labels match the design
• Confirm there's exactly one record per unit per variant
• Check date range matches the experiment window
• Strip any users who appeared in multiple variants (assignment errors)

1. Run Sample Ratio Mismatch (SRM) check.

• Compute observed vs expected ratio
• χ² test against design allocation
• If p < 0.001, stop. SRM means broken assignment — results are invalid.

1. Compute primary metric per variant.

• Point estimate
• 95% confidence interval (use bootstrap for ratio metrics)
• Absolute lift and relative lift vs control

1. Test significance.

• Proportion: two-proportion z-test
• Mean: Welch's t-test
• Ratio: delta method or bootstrap
• Report p-value AND CI (CI is more useful than p-value alone)

1. Evaluate guardrails. Each guardrail gets one of three statuses:

• PASS — within threshold with 95% confidence
• WATCH — point estimate breaches but CI overlaps zero
• FAIL — CI clears the threshold in the wrong direction

1. Decision recommendation. Apply this matrix:

1. Write the readout using the template below.

Output format

# Experiment Readout: <name>

## TL;DR
**Recommendation:** <Ship | Hold | Kill | Inconclusive>

<Primary metric> changed by <X%> ([<low>, <high>] 95% CI, p=<value>) over <N> days.

## Sample Ratio Mismatch
| Variant | Expected | Observed | Delta |
|---|---|---|---|
| Control | 50.0% | 49.8% | -0.2pp |
| Treatment | 50.0% | 50.2% | +0.2pp |

χ² p-value: <value> — **<PASS|FAIL>**

## Primary metric: <name>
| Variant | N | Value | 95% CI |
|---|---|---|---|
| Control | <N> | <val> | [<low>, <high>] |
| Treatment | <N> | <val> | [<low>, <high>] |

- Absolute lift: <X> ([<low>, <high>])
- Relative lift: <X%> ([<low>, <high>])
- Test: <z-test | t-test | bootstrap>, p = <value>

## Guardrails
| Metric | Threshold | Result | Status |
|---|---|---|---|
| <metric> | <threshold> | <delta + CI> | <PASS|WATCH|FAIL> |

## Subgroup analysis (pre-specified only)
<table or "No pre-specified subgroups">

## Decision
<Ship | Hold | Kill | Inconclusive>, because <reason>.

## Caveats
- <pre-registration deviations if any>
- <data quality notes>
- <known confounds>

## Next steps
- <action 1>
- <action 2>

Validation checks

• [ ] Total sample reached the pre-registered target
• [ ] SRM check ran and passed
• [ ] Confidence interval reported (not just p-value)
• [ ] All pre-registered guardrails evaluated
• [ ] Any subgroup analysis flagged as pre-specified vs exploratory
• [ ] Recommendation matches the decision matrix

Edge cases & failure modes

• Peeking: if the user asks for an analysis before planned sample size, refuse to make a ship decision. Compute confidence intervals only and label "interim — do not act on this."
• One-sided tests: convert to two-sided unless pre-registered as one-sided. Two-sided is the default.
• Multiple primary metrics: if there are more than one, you cannot ship on "any" being significant. Either pick one or apply Bonferroni correction.
• Heavy-tailed metrics (revenue): trimmed mean (winsorize at 99th percentile) is often more robust than raw mean. Note this in caveats.
• Day-of-week effects: if runtime isn't a multiple of 7 days, day-of-week imbalance can bias results. Note in caveats.

Scripts

• scripts/analyze_experiment.py — End-to-end analysis from a CSV of unit_id, variant, metric_value.

python scripts/analyze_experiment.py \
    --input results.csv \
    --metric-type proportion \
    --control-label control \
    --treatment-label treatment

Related skills

• ab-test-design — pre-experiment planning
• metric-definition — clarify guardrail thresholds
• stakeholder-readout — for non-experiment analyses needing similar structure
• causal-inference — when randomization is broken or unavailable