hypothesis-first — independently scanned and version-tracked by SaferSkills.
SaferSkills independently audited hypothesis-first (Agent Skill) and scored it 100/100 (green). The audit ran 55 deterministic rules across Security, Supply Chain, Maintenance, Transparency, and Community; it found 0 high-severity and 0 lower-severity findings. The full rule-by-rule trace and per-finding evidence are below. Free, methodology-open.
Findings & checks · 0 flagged
Every scanned point with the score it earned and what moved between them.
First recorded scan — no prior version to compare against.
The primary manifest — the file an agent reads to learn what this artifact does.
Register the hypothesis first. Run the analysis. Interpret against the prediction.
Core principle: If you didn't commit your hypothesis before seeing the data, you don't know if you're doing science or storytelling.
Violating the letter of the rules is violating the spirit of the rules.
Always:
Exceptions (discuss with your PI or collaborator):
Thinking "skip registration just this once"? Stop. That's rationalization.
NO DATA ANALYSIS WITHOUT A REGISTERED HYPOTHESIS FIRSTRan the analysis before registering? Delete the results. Start over.
No exceptions:
Register first, then run. Period.
digraph hypothesis_first_cycle {
rankdir=LR;
register [label="REGISTER\nState H1, H0, outcome,\ntest, threshold.", shape=box, style=filled, fillcolor="#ffcccc"];
verify_register [label="Inline verify\nregistration\nis complete", shape=diamond];
subagent_review [label="Dispatch\nregistration-reviewer\n(fresh subagent)", shape=box, style=filled, fillcolor="#ffaaaa"];
commit [label="git commit\n(only if Approved)", shape=box, style=filled, fillcolor="#ff8888"];
execute [label="EXECUTE\nRun pre-specified\nanalysis exactly.\nNo deviations.", shape=box, style=filled, fillcolor="#ccffcc"];
verify_execute [label="Any deviation\nfrom plan?", shape=diamond];
document_deviation [label="Document as\nEXPLORATORY\nlabel deviation", shape=box, style=filled, fillcolor="#ffffcc"];
interpret [label="INTERPRET\nCompare to prediction.\nReport ALL results.\nNo selective reporting.", shape=box, style=filled, fillcolor="#ccccff"];
next [label="Next\nhypothesis", shape=ellipse];
register -> verify_register;
verify_register -> subagent_review [label="complete"];
verify_register -> register [label="incomplete:\nfinish it"];
subagent_review -> register [label="Issues Found:\nfix and re-review"];
subagent_review -> commit [label="Approved"];
commit -> execute;
execute -> verify_execute;
verify_execute -> document_deviation [label="yes"];
verify_execute -> interpret [label="no"];
document_deviation -> interpret;
interpret -> next;
next -> register;
}This is the equivalent of writing a failing test. You commit to what you expect before you can possibly know whether you are right.
A complete registration contains ALL of the following:
docs/references/statistical-guide.md for test selection.)docs/references/data-checklist.md §4).cohort-v2.4.0-2024-03-15), file hash(es) (SHA256 of every input file the analysis will read), and preprocessing pipeline version (git commit hash of the preprocessing script). This is the scientific equivalent of a package-lock file — without it, "same analysis" is not reproducible. See docs/references/data-checklist.md §7.Dispatching the Registration Reviewer (PRE-COMMIT GATE — NON-NEGOTIABLE):
Before running git add or git commit, dispatch a fresh subagent reviewer to verify the registration is complete, specific, and consistent with the approved design document. This is the strictest gate in Eureka. Registrations are immutable after commit — any change after commit counts as HARKing or undisclosed protocol deviation. This reviewer is your last line of defense against permanent scientific debt.
skills/hypothesis-first/registration-reviewer-prompt.md{REGISTRATION_PATH} → the draft registration file path (not yet committed){DESIGN_DOC_PATH} → the approved design document from research-brainstorminggeneral-purpose subagent) with the filled promptStatus: Approved or Status: Issues FoundActing on the reviewer's response:
git add and git commit belowApproved. There is no exception to this rule — committing an unreviewed registration, or committing a registration the reviewer rejected, produces permanent scientific debt that cannot be retracted without a HARKing disclosure.If the reviewer rejects the same issue twice after attempted fixes, escalate to the user: describe the issue, the attempted fix, and ask for guidance. Do not commit.
What "committed to version control" means concretely (only after reviewer approval):
# Create the registration file
docs/eureka/registrations/YYYY-MM-DD-<study-id>-registration.md
# Dispatch the registration-reviewer subagent FIRST (see above)
# Only after Status: Approved, proceed with:
# Stage and commit BEFORE running any analysis
git add docs/eureka/registrations/YYYY-MM-DD-<study-id>-registration.md
git commit -m "Register hypothesis: <one-line summary>"
# Record the commit hash in your lab notebook or results file
git rev-parse HEADThe timestamp on that commit is your proof. It exists before your results exist. If the commit and the results file share a timestamp, you did not pre-register.
Verify the registration is complete:
Not complete? Go back. Do not touch the data.
Run the pre-specified analysis. Exactly.
Requirements:
If something goes wrong:
| Situation | Correct Response |
|---|---|
| Data quality issue discovered | Stop. Document it. Decide whether to proceed per pre-specified exclusion rules. If not covered, note it as a protocol deviation. |
| Assumption violated (e.g., normality) | Only switch tests if your registration specified a contingency (e.g., "if Shapiro-Wilk p < 0.05, use Spearman"). If not specified, this is a deviation — document and label exploratory. |
| Better analysis idea mid-run | Stop. Do not run it yet. Finish the registered analysis first. Register the new idea as a separate hypothesis. |
| Software error forces a change | Document the error, the forced change, and why it was unavoidable. Label affected results as a protocol deviation. |
Deviations are allowed. Undisclosed deviations are fraud.
Every deviation from the pre-specified plan must be:
You have results. Compare them to what you said you expected. Report everything.
Requirements:
Minimum Reporting Standard (every result):
No statistical claim is complete without ALL of these:
Template: [Test]: [statistic]([df]) = [value], p = [exact], [effect size] = [value]
(95% CI: [lower, upper]), n = [per group]. [Correction] for [N] comparisons.See docs/references/statistical-guide.md for test selection and effect size interpretation.
What "compare to prediction" means:
Your registration said: "r > 0.4, one-tailed, α = 0.05." Your result is: r = 0.31, p = 0.09.
This is a null result. Report it as a null result. Do not:
Null results are results. A null result from a pre-registered hypothesis is more informative than a significant result from a HARKed one. Null results are publishable. Null results protect the field from wasted replications.
Selective reporting — reporting only significant results — is p-hacking by omission. It inflates the published literature's false positive rate just as much as running 20 analyses and reporting the one that worked.
| This is NOT registration | This IS registration |
|---|---|
| "I have a hypothesis in my head" | Hypothesis written, committed to VCS before analysis |
| "My PI and I discussed it" | Design document written, committed to VCS before analysis |
| "I wrote it in my notebook after seeing the trend" | Commit timestamp predates analysis run timestamp |
| "It was obvious that's what we'd test" | H1, H0, test, threshold explicitly stated in writing |
| "I registered it on OSF after running the analysis" | Registration timestamp demonstrably precedes data access |
| "I'll document the hypothesis in the methods section" | Methods written post-hoc describe what happened, not what was predicted |
| Excuse | Reality |
|---|---|
| "My hypothesis is obvious — everyone would predict this" | Unstated hypotheses get adjusted after results. Write it down. |
| "I just want to look at the data first" | Looking at the data before locking the analysis plan is how p-hacking starts. Every time. |
| "I'll register after I confirm the analysis is feasible" | Feasibility checks count as data analysis. Register what you'll check in the feasibility check. |
| "Pre-registration is for clinical trials, not basic research" | HARKing inflates Type I error regardless of research domain. The biology does not care about your field's norms. |
| "We're in exploratory mode — no need to pre-register" | Exploratory work is fine. Label it as exploratory. Do not later present it as confirmatory without a replication. |
| "The analysis plan will change as I understand the data better" | That is post-hoc analysis. You may do it. Label it as post-hoc. Do not present it as hypothesis-driven. |
| "I've run this analysis before — I know what to expect" | Prior experience informs your registered prediction. Write down that prediction. |
| "Pre-registration slows down the research" | Writing the registration takes 30 minutes. Defending a p-hacking accusation in peer review takes months. |
| "The effect is so strong it doesn't need pre-registration" | Strong effects replicate. If your effect is real, it will survive a pre-registered replication. Register it. |
| "I just want to run a quick preliminary analysis" | Preliminary analyses become the paper. The "quick look" becomes Figure 1. Design first. |
| "We can always replicate later to confirm" | Most replication attempts never happen. Your first analysis is likely the record. |
| "Nobody actually pre-registers in this field" | Reproducibility crises happen in fields where nobody pre-registers. You are looking at cause and effect. |
These situations mean stop and either register properly or explicitly label as exploratory:
All of these mean: Delete the analysis results. Restart from REGISTER.
Kept the unregistered results as "reference"? Delete them. Looked at effect sizes to calibrate your prediction? Your registration is compromised. Start from a new hypothesis on a held-out sample if one exists, or label the entire analysis as exploratory.
Before treating any analysis as confirmatory:
registration-reviewer subagent dispatched and returned Status: ApprovedCannot check all boxes? The analysis is exploratory. Label it as such. Do not present it as hypothesis-driven.
Exploratory analysis is not forbidden. It is essential. The rule is not "no exploration" — the rule is "no mislabeling."
Exploratory analysis is legitimate when:
The exploratory-to-confirmatory pipeline:
digraph exploratory_confirmatory {
explore [label="Exploratory Analysis\n(labeled as such)", shape=box, style=filled, fillcolor="#ffffcc"];
generate [label="Generate hypothesis\nfrom finding", shape=box];
register [label="REGISTER hypothesis\non new/held-out sample", shape=box, style=filled, fillcolor="#ffcccc"];
execute [label="EXECUTE\npre-specified analysis", shape=box, style=filled, fillcolor="#ccffcc"];
interpret [label="INTERPRET\nconfirmatory result", shape=box, style=filled, fillcolor="#ccccff"];
explore -> generate;
generate -> register [label="new sample\nor split-half"];
register -> execute;
execute -> interpret;
interpret -> explore [label="generates new\nexploratory questions"];
}Found something in exploratory analysis and want to claim it as a result? Replicate it in a pre-registered study. No exceptions.
eureka:research-brainstorming — invoked at the end of the design phase to lock the hypothesis before any analysis beginsdocs/references/statistical-guide.md — test selection, effect size interpretation, correction methodsdocs/references/data-checklist.md — data version locking, preprocessing requirements, missing value handlingdocs/references/registration-lifecycle.md — YAML frontmatter schema, filename convention, amendment vs supersede decision tree, HARKing severity spectrum, data-discovery feedback workflowIf eureka:research-brainstorming completed a design document, the registration here formalizes it into version control. The design document is the draft; the VCS commit is the registration.
Pre-registration does NOT mean registrations are frozen. Studies discover that assumptions were wrong, data came out different, or the hypothesis needs revision. docs/references/registration-lifecycle.md covers the full lifecycle:
status: active | amended-by | superseded-by | archived — machine-readable lifecycle stateYYYY-MM-DD-<topic>-registration.md, amendments YYYY-MM-DD-<topic>-amendment-NNN.md, versioned registrations YYYY-MM-DD-<topic>-v<N>-registration.mddocs/eureka/registrations/INDEX.md is the machine-readable chain of all registrations. Every time this skill creates, amends, or supersedes a registration, regenerate the affected rows in INDEX.md. Without INDEX.md, 3 months after a pivot nobody knows which registration is authoritative — this was a concrete failure mode reported in external feedback.
The skill's workflow now includes:
docs/eureka/registrations/INDEX.md with the new row and affected lineageregistration-reviewer subagent (existing step)registration-lifecycle.md formalizes the spectrum from "observation noted" (no concern) to "silent registration edit" (CRITICAL fraud-adjacent). Use this spectrum to judge severity of any deviation the researcher discovers mid-study — the default mental model should NOT be "all deviations = HARKing" (which pushes hiding) but "specific severity tier determines disclosure requirement".
RIGID — Follow exactly. Do not adapt away the discipline.
The requirement to commit a hypothesis before running an analysis is not a guideline to balance against convenience. It is the line between confirmatory and exploratory science. Every rationalization that has ever appeared in a retracted paper sounded reasonable at the time. The rule exists precisely because the rationalizations are convincing.
If the rule seems too strict for your situation: label your analysis as exploratory and proceed. That is the correct response. "This is exploratory" is always available. "This is confirmatory but I didn't pre-register" is never available.
~30 seconds. Free. No account. Every finding cites a rule and a line of evidence.