aaai-reproducibility — independently scanned and version-tracked by SaferSkills.
SaferSkills independently audited aaai-reproducibility (Agent Skill) and scored it 100/100 (green). The audit ran 55 deterministic rules across Security, Supply Chain, Maintenance, Transparency, and Community; it found 0 high-severity and 0 lower-severity findings. The full rule-by-rule trace and per-finding evidence are below. Free, methodology-open.
Findings & checks · 0 flagged
Every scanned point with the score it earned and what moved between them.
First recorded scan — no prior version to compare against.
The primary manifest — the file an agent reads to learn what this artifact does.
Use this when a draft needs to survive AAAI review on rigor, not just novelty. AAAI-27 requires the reproducibility checklist to be uploaded separately from the main PDF, in its own field on the submission form (AAAI-26 carried it inside the PDF after the references) — so it is a document a reviewer opens on its own, and it has to agree with the paper and supplement rather than read as an afterthought. AAAI-27 also states that reviewers assess reproducibility from what was actually submitted, and that material promised "after acceptance or publication" is not evidence it exists.
checklist answer, or code/data artifact.
selection, and hardware.
annotation procedures.
AAAI places the reproducibility checklist after the references, and reviewers cross-check each "yes" against the paper and supplement. A "yes" with no backing artifact reads worse than an honest "no", because it signals the checklist was filled in carelessly.
| Checklist answer | Must be backed by | Phase-1 risk if unbacked |
|---|---|---|
| code available | runnable scripts in the ZIP | "claimed but absent" |
| seeds reported | seed list and variance | "single-run cherry-pick" |
| compute disclosed | train vs. inference vs. search cost | "hidden tuning budget" |
| data accessible | license and access path | "irreproducible by anyone" |
Create a row for every claim that appears in the abstract, introduction, or conclusion. The ledger should be short enough to audit before submission and concrete enough that a Phase-1 reviewer can see that each headline claim is checkable.
| Ledger field | What to record | Common failure |
|---|---|---|
| Claim text | exact sentence or paraphrase from the paper | claim becomes stronger than the evidence |
| Evidence artifact | theorem, table, figure, appendix, code command, data sheet, or log path | evidence exists but is not submitted |
| Reproducibility inputs | seeds, splits, prompts, preprocessing, hardware, hyperparameters, and model versions | rerun cannot recreate the result |
| Variance and controls | confidence interval, standard deviation, multiple seeds, ablation, or matched-compute baseline | single lucky run drives the claim |
| Checklist answer | the checklist item whose answer depends on this artifact | checklist contradicts the supplement |
| Reviewer risk | what a skeptical reviewer would challenge first | rebuttal cannot fix missing evidence |
For each row, choose one of three actions: keep the claim because the artifact is present, weaken the claim to match the evidence, or add the missing artifact before submission. Do not leave a row in "promise later" state.
Before upload, run the artifact as if the reviewer has no private context:
are stated before the command.
The dry-run can be small; it does not need to reproduce every experiment. Its purpose is to prove that the submitted artifact is not merely decorative and that the checklist answers are honest.
before the deadline; do not promise the repository in rebuttal.
answer to match the supplement exactly.
the claim is checkable.
A vision-language paper checks "code and data available" but the ZIP holds only PDFs of plots. Audit verdict: reproducibility grade "fragile", with a checklist conflict between the "yes" and the missing scripts. The smallest fix is a reproduce.sh that regenerates one headline table from seeds plus a dataset license note, after which the checklist answer becomes truthful and Phase-1 defensible.
[Reproducibility grade] strong / adequate / fragile / not reviewable
[Checklist conflicts] <answers that contradict paper/supplement>
[Evidence gaps] <claims without submitted verification>
[Compute/data disclosure] complete / incomplete
[Priority fixes] <smallest changes before submission>~30 seconds. Free. No account. Every finding cites a rule and a line of evidence.