os-experiment-log — independently scanned and version-tracked by SaferSkills.
SaferSkills independently audited os-experiment-log (Agent Skill) and scored it 87/100 (green). The audit ran 55 deterministic rules across Security, Supply Chain, Maintenance, Transparency, and Community; it found 0 high-severity and 3 lower-severity findings. The full rule-by-rule trace and per-finding evidence are below. Free, methodology-open.
Findings & checks · 3 flagged
The text {match} tells the agent to skip the normal "ask the user first" gate. Used adversarially it removes the human-in-the-loop check before destructive or sensitive actions, turning a normally-gated agent into a fire-and-forget executor.
Every scanned point with the score it earned and what moved between them.
First recorded scan — no prior version to compare against.
The primary manifest — the file an agent reads to learn what this artifact does.
The experiment log is the unified cross-cutting record for all agentic-os experiments. One file per run, all files in context/experiment-log/, with index.md as a queryable table of all runs.
context/experiment-log/
index.md ← one row per run (date, source, target, verdict)
2026-04-25-verifier-os-architect-round1.md ← from os-evolution-verifier
2026-04-25-tester-os-architect.md ← from os-architect-tester
2026-04-25-os-improvement-loop-os-eval-runner.md ← from os-improvement-loop
2026-04-25-planner-0024.md ← from os-evolution-planner
2026-04-25-survey-session.md ← from post_run_surveyAgents must check result_type in a log entry's header before parsing it:
--source-type | Produced by | result_type | Key fields |
|---|---|---|---|
verifier | os-evolution-verifier | qualitative | PASS/PARTIAL/FAIL counts, HANDOFF_BLOCK validity |
tester | os-architect-tester | qualitative | AC-1–4 pass/fail per scenario |
orchestrator | os-improvement-loop | numeric | best_score, baseline, delta, KEEP/DISCARD counts |
planner | os-evolution-planner | qualitative | workstream count, gaps identified |
survey | post_run_survey | mixed | friction item count, north_star metric |
Numeric entries (result_type: numeric) carry quantitative metrics suitable for trending and charting. Qualitative entries (result_type: qualitative) carry pass/fail verdicts and gap analysis prose. Mixed entries (result_type: mixed) carry both — agents must check which fields are present before parsing.
Read the argument or invocation context to determine mode:
context/experiment-log/ by keyword# After os-evolution-verifier run
python3 scripts/experiment_log.py append \
--source-type verifier \
--report temp/os-evolution-verifier/test-report.md \
--session-id 2026-04-25-round1 \
--target os-architect \
--triggered-by os-evolution-verifier
# After os-architect-tester run
python3 scripts/experiment_log.py append \
--source-type tester \
--report temp/test_report_consolidated.md \
--session-id 2026-04-25-tester \
--target os-architect \
--triggered-by os-architect-tester
# After os-improvement-loop run (numeric — has score delta)
python3 scripts/experiment_log.py append \
--source-type orchestrator \
--report temp/logs/run-log.md \
--session-id 2026-04-25-os-eval-runner \
--target os-eval-runner \
--triggered-by os-improvement-loop
# After os-evolution-planner writes a task plan
python3 scripts/experiment_log.py append \
--source-type planner \
--report tasks/todo/0024-plan.md \
--session-id 0024 \
--target os-eval-runner \
--triggered-by os-evolution-planner
# After a post-run survey
python3 scripts/experiment_log.py append \
--source-type survey \
--session-id 2026-04-25-session \
--target session \
--triggered-by human
# Query by term
python3 scripts/experiment_log.py query T2-D
python3 scripts/experiment_log.py query FAIL
python3 scripts/experiment_log.py query numeric
# Aggregate summary
python3 scripts/experiment_log.py summaryAfter append:
tail -5 context/experiment-log/index.mdReport: "Logged to context/experiment-log/<filename>. Index updated."
After query: relay matching file names and their header blocks (date, source, target, verdict).
After summary: print the per-source-type breakdown verbatim.
Each file has a YAML-like header fence followed by the full report:
---
type: verifier
result_type: qualitative
date: 2026-04-25 15:12
session_id: 2026-04-25-round1
source: os-evolution-verifier
target: os-architect
verdict: 8P/0Pa/0F of 8
---
## Experiment — 2026-04-25 15:12 | verifier | os-architect
| Field | Value |
...
[full report content]
### Actions Taken
_[fill in: spec fix, new eval, new skill]_Smoke 1 — Append verifier: Run python3 scripts/experiment_log.py append --source-type verifier. Confirm new .md file appears in context/experiment-log/ and index.md has a new row.
Smoke 2 — Query: Run python3 scripts/experiment_log.py query PASS. Confirm output lists at least one matching file with its header.
Smoke 3 — Summary by type: Run python3 scripts/experiment_log.py summary. Confirm output shows [verifier], [orchestrator] etc. sections with correct run counts.
survey source typecontains both friction prose and numeric north_star values. Always check result_type in the file header before assuming field presence.
append immediately after a run completes, before anyshell restart. The script exits with an error if the report file is missing rather than appending empty data.
without response actions is an audit trail, not a learning record. Fill it in before the next run.
append is called twice for the same session, two rowsappear in index.md. This is intentional (the file is append-only) but worth noting when querying.
~30 seconds. Free. No account. Every finding cites a rule and a line of evidence.