experimentation — independently scanned and version-tracked by SaferSkills.
SaferSkills independently audited experimentation (Agent Skill) and scored it 100/100 (green). The audit ran 55 deterministic rules across Security, Supply Chain, Maintenance, Transparency, and Community; it found 0 high-severity and 0 lower-severity findings. The full rule-by-rule trace and per-finding evidence are below. Free, methodology-open.
Findings & checks · 0 flagged
Every scanned point with the score it earned and what moved between them.
First recorded scan — no prior version to compare against.
The primary manifest — the file an agent reads to learn what this artifact does.
Shipping a change and watching the metric move is not evidence — the metric moves for a dozen reasons, and humans are superb at seeing the result they wanted. An experiment is the discipline that separates the change worked from the number went up: state the hypothesis and the metric before you look, size the test so the result is real, watch guardrails for collateral damage, and read it out honestly — including "no effect" and "it got worse."
This closes the loop the rest of the lifecycle opens: [[product-discovery]] defines the success metric, [[feature-flags]] delivers the variants, [[observability]] instruments them, and experimentation decides whether to keep, kill, or iterate. It pairs with the growth skills when the test runs on a campaign or funnel ([[paid-ads]], [[growth-strategy]]).
Skip when there's no measurable outcome (internal refactor, bug fix with a clear correct answer), or when traffic is far too low for any test to reach significance — then decide by judgment and say so, don't fake a test. For a correctness check use [[test-first]] / [[browser-checks]]; experimentation is about impact, not whether the code runs.
Not a substitute for [[observability]] (the instrumentation it relies on) or [[product-discovery]] (which defines which metric matters).
State it as a falsifiable prediction: "Changing X will improve [primary metric] by ~Y% because [mechanism]." No mechanism = a fishing trip. Write it down now so you can't move the goalposts after seeing data.
p-hack your way to a "win."
unsubscribes). A conversion lift that tanks retention is a loss. Tie both to real instrumentation ([[observability]]); a metric you can't measure cleanly isn't a metric.
Decide the minimum detectable effect you care about, pick significance and power (commonly 95% / 80%), and compute the sample size needed — then translate to days of traffic. Running until it "looks significant" and stopping (peeking) manufactures false wins. Fix the stop in advance, or use a method built for sequential looks.
Randomize at the right unit (usually user, not request, or you'll split one person across variants). Verify the split is balanced and that there's no leakage between arms ([[feature-flags]] is the clean delivery mechanism). A broken randomization invalidates everything downstream — check it before trusting results.
Let it run to the pre-committed sample/duration. Monitor guardrails for safety (kill if something breaks — that's allowed), but don't call the result early because it's trending your way. Watch for novelty/primacy effects on short tests.
significant" is a real, useful outcome — it means don't ship on this evidence, not "try until it wins."
Ship, kill, or iterate — and write down the decision, the numbers, and the reasoning ([[decision-docs]]). A killed experiment that's documented is a saved future mistake. Then clean up the flag and losing variant ([[feature-flags]] debt).
~30 seconds. Free. No account. Every finding cites a rule and a line of evidence.