Pre-registration Discipline (Iron Law)

Skill type: DISCIPLINE-ENFORCING. This is not a how-to and not a reference. It exists to stop one specific failure mode — exploiting researcher degrees of freedom after seeing the data, then reporting the result as if it were planned. It governs when you are allowed to claim a confirmatory result; it does not teach you how to pick a test or fill an OSF form. Those are sibling skills (see the routing table).

THE IRON LAW

    NO DATA ANALYSIS WITHOUT A PRE-REGISTERED ANALYSIS PLAN FIRST.

The frozen hypothesis + analysis plan is the research analog of a failing test written before the implementation: you commit to what would count as a result before you can see whether you got one. Run an unplanned analysis and the rule is absolute — it is exploratory. Label it, timestamp it, and you cannot report its p-value as confirmatory. (Per the Center for Open Science confirmatory/exploratory model: in exploratory work, p-values lose their diagnostic value and findings require independent replication. Verified at https://www.cos.io/initiatives/prereg.)

Violating the letter of the pre-registration is violating the spirit of the science.

That line pre-empts the predictable defense — "pre-registration is bureaucratic, I'm following the scientific spirit." There is no spirit-following exception. Iterate freely in the EXPLORE phase; the confirmatory claim needs the frozen plan.

When to Use This Skill

Use it the moment a confirmatory claim is on the table without a frozen plan behind it, or the moment the plan is being bent after data are visible:

• About to analyze a dataset and there is no written, pre-data analysis plan.
• Switching the primary outcome, adding/removing a covariate, or changing the test

after seeing results or p-values.

• Peeking at accruing data and weighing whether to stop now ("it's already significant").
• Dropping outliers, transforming variables, or changing exclusions that were not

pre-specified.

• Drafting an OSF / AsPredicted / PROSPERO pre-registration (route the mechanics to

alterlab-open-science; this skill enforces that the plan is actually frozen and honored).

• Rationalizing a deviation from an existing registration.
• A deadline-pressure ask like *"just p-hack this borderline result into something

publishable."* — refuse the framing; offer the PLAN/CONFIRM/EXPLORE path instead.

Does NOT Trigger

This skill is the gatekeeper, not the toolbox. It orchestrates siblings and routes everything mechanical away. Route adjacent requests as follows:

REQUIRED BACKGROUND (this skill orchestrates, it does not reimplement): alterlab-open-science for the registration artifact, alterlab-statistical-analysis for test choice + assumptions, alterlab-scientific-thinking for bias framing. When a step needs any of those, hand off — do not restate their content here.

The Workflow: PLAN → COLLECT → CONFIRM → EXPLORE

This is the research analog of red → green → refactor. Each phase has an exit gate; you may not enter the next phase until the current gate passes.

1. PLAN — freeze it before any data are visible

Write down, and lock, all of:

1. The hypotheses (directional where applicable).
2. The primary outcome and any secondary outcomes — ranked.
3. The exact test(s) for each hypothesis (defer the choice to

alterlab-statistical-analysis; record the decision here).

1. Inclusion/exclusion rules and the outlier rule, pre-specified.
2. The stopping rule / planned sample size (so optional stopping is off the table).
3. Covariates, transformations, and how missing data are handled.

Register the artifact via alterlab-open-science (OSF Registries / AsPredicted / PROSPERO). Gate: the plan exists, is timestamped, and nothing in it depends on having seen the outcome data. If you cannot answer "what result would falsify this?" you are not done planning.

2. COLLECT — data, untouched

Collect according to the stopping rule. Do not peek at the outcome to decide whether to keep going. If an interim look is genuinely needed, it had to be a pre-specified sequential design (record that in PLAN). Gate: data collection matched the frozen rule; no outcome-dependent stopping occurred.

3. CONFIRM — run exactly the planned tests

Run the pre-specified tests, in the pre-specified order, on the pre-specified sample. Assumption checks (via alterlab-statistical-analysis) run and are reported before interpreting the result — you do not get to swap the test because an assumption failed unless the swap was pre-specified; an unplanned swap demotes the result to EXPLORE. Gate: every confirmatory number traces to a line in the frozen plan.

4. EXPLORE — everything else, explicitly flagged

Anything not in the frozen plan lives here: new subgroups, post-hoc covariates, alternative tests, interesting patterns. This work is valuable — it generates the next study's hypotheses — but it is reported under an Exploratory heading, its p-values are descriptive not confirmatory, and it carries a "requires replication" caveat. Gate: every exploratory finding is labeled as such; none is laundered into the confirmatory narrative.

A decision flowchart and the full phase gates live in references/workflow_gates.md.

Excuse vs. Reality

The predictable rationalizations, and what is actually happening. (Seeded from the documented researcher-degrees-of-freedom literature — see references/researcher_degrees_of_freedom.md.)

Red Flags — STOP

If you catch yourself (or the user) thinking any of these, STOP:

• "Let me just try a different test and see."
• "I'll drop these outliers and rerun."
• "The effect is there if I add this one covariate."
• "We can stop collecting now — it's already significant."
• "I'll just report the analyses that worked."
• "This subgroup is fascinating (we didn't predict it, but…)."
• "Let me change the primary outcome to the one that came out."
• "Pre-registration is overkill for a study like this."

All of these mean the same thing: you are exploiting researcher degrees of freedom. Either return to the frozen plan, or label the work exploratory and forfeit the confirmatory claim. There is no third option.

The Multiple-Comparisons Escalation Gate

A hard, countable rule (the analog of "3 failed fixes = wrong architecture"):

Ran 3+ tests on the same hypothesis searching for significance? STOP. This is multiple comparisons / test-shopping. Either correct for all of them (Bonferroni / Benjamini–Hochberg FDR — via alterlab-statistical-analysis) or declare the whole analysis exploratory. Do not run test #4 to find p < .05.

Deviation Disclosure Gate

If a deviation from the frozen plan is unavoidable (a genuine error in the plan, an impossible assumption), it is not silently absorbed. Run, in order:

1. STATE the original frozen specification (quote the plan).
2. STATE the deviation and the concrete reason it was forced.
3. CLASSIFY the affected result as exploratory unless the deviation provably

cannot have been outcome-driven.

1. DISCLOSE the deviation in the manuscript's transparency/deviations section

(route reporting completeness to alterlab-results-transparency).

You can run a quick structured self-audit of a plan-vs-actual story with the optional helper:

uv run python skills/methodology/alterlab-preregistration-discipline/scripts/prereg_check.py \
    --plan plan.json --actual actual.json

It is a stdlib-only checklist scorer (no network, no third-party APIs) — it does not run statistics. See its --help.

Self-Check Before Claiming a Confirmatory Result

• Does a frozen, timestamped plan exist, written before the outcome data were visible?
• Does every confirmatory number trace to a line in that plan?
• Were assumption checks run and reported before interpretation?
• Is every unplanned analysis under an Exploratory heading with a replication caveat?
• Are all deviations disclosed, not absorbed?

Any "no" → the result is exploratory, not confirmatory. Say so plainly.

References

• references/workflow_gates.md — the PLAN/COLLECT/CONFIRM/EXPLORE

phase gates and decision flowchart in full.

• references/researcher_degrees_of_freedom.md —

the documented failure modes (HARKing, optional stopping, p-hacking, the garden of forking paths) this skill is built to catch, with sources.

• Center for Open Science — Preregistration. https://www.cos.io/initiatives/prereg

(confirmatory vs. exploratory model; verified during authoring).

• Orchestrated siblings: alterlab-open-science, alterlab-statistical-analysis,

alterlab-scientific-thinking.

Part of the AlterLab Academic Skills suite.