test-napkin-math — independently scanned and version-tracked by SaferSkills.
SaferSkills independently audited test-napkin-math (Agent Skill) and scored it 100/100 (green). The audit ran 55 deterministic rules across Security, Supply Chain, Maintenance, Transparency, and Community; it found 0 high-severity and 0 lower-severity findings. The full rule-by-rule trace and per-finding evidence are below. Free, methodology-open.
Findings & checks · 0 flagged
Every scanned point with the score it earned and what moved between them.
First recorded scan — no prior version to compare against.
The primary manifest — the file an agent reads to learn what this artifact does.
A single-shot smoke check for the napkin_math experiment. The test logic lives in experiments/napkin_math/tests/run_smoke.py; this skill is a thin wrapper that invokes the script, parses its output, and reports a one-line summary.
Covers the Monte Carlo runner (end-to-end, determinism, Bernoulli arithmetic, sensitivity ranking), the strict-schema fail-fast paths (each required field individually), prepare_extract_input.py import sanity, and the compress_report_section pytest suite.
experiments/napkin_math/run_monte_carlo.pyexperiments/napkin_math/prepare_extract_input.pyexperiments/napkin_math/tests/fixtures/smoke/*worker_plan/worker_plan_internal/parameter_extraction/compress_report_section.pyexperiments/napkin_math/.claude/skills/{extract-parameters-from-full,extract-parameters-from-digest,generate-bounds,generate-calculations,run-scenarios,monte-carlo}/ that touches the artifact schemaNot for: end-to-end LLM-driven runs of the extract-parameters-from-full skills against real reports (those require an LLM in the loop and are out of scope here).
/opt/homebrew/bin/python3.11 experiments/napkin_math/tests/run_smoke.pyOverride the interpreter with NAPKIN_TEST_PYTHON=<path> if the default isn't available.
ok / FAIL lines, then a final summary: SUMMARY: 7/7 checks passed
ALL GREENExit code is 0 on full pass, 1 on any failure.
| Check | What it verifies |
|---|---|
end_to_end | Runner consumes the synthetic fixture, emits both expected outputs, zero warnings |
determinism | Two runs with the same seed produce byte-identical JSON |
bernoulli_arithmetic | Mean of total_budget_with_gate_inr ≈ 1,000,000 + 0.6 * 250,000 within ±5,000 |
sensitivity_ranking | Bernoulli gate is the sole driver of its dependent output (correlation == 1.0); two-input convert sensitivity has both inputs |
schema_errors | Each required schema field (sampling_discipline, non_negative, default_pass_probability for bernoulli_gate, output_name, output_unit) triggers a SCHEMA ERROR exit code 2 with a message naming the upstream stage to re-run |
prepare_extract_input_imports | prepare_extract_input.py imports cleanly and exposes build_combined_digest |
compress_pytest | The pytest suite for compress_report_section.py passes (13 tests) |
| Mistake | Fix |
|---|---|
| Skipping the check because "the diff is tiny" | The schema is tightly coupled across five skills + the runner. A tiny change in one prompt can break the runner's strict validation. Always run it. |
| Reporting "tests pass" without running the script | If you didn't see ALL GREEN, you didn't pass. |
Running with python3 instead of python3.11 | NumPy and the worker_plan tests live in the 3.11 env. Use the explicit path or set NAPKIN_TEST_PYTHON. |
Treating a compress_pytest failure as unrelated | The compressor is part of the napkin_math experiment; if its tests break, the digest produced by prepare_extract_input.py is suspect. |
| Editing fixtures to make tests pass | Fixtures encode the contract. If a fixture needs to change, the schema or the runner changed for a real reason — update both consciously and re-justify each touched check. |
experiments/napkin_math/tests/run_smoke.pyexperiments/napkin_math/tests/fixtures/smoke/experiments/napkin_math/run_monte_carlo.py../extract-parameters-from-full/SKILL.md, ../extract-parameters-from-digest/SKILL.md, ../generate-bounds/SKILL.md, ../generate-calculations/SKILL.md, ../run-scenarios/SKILL.md, ../monte-carlo/SKILL.md~30 seconds. Free. No account. Every finding cites a rule and a line of evidence.