causality-protocol — independently scanned and version-tracked by SaferSkills.
SaferSkills independently audited causality-protocol (Agent Skill) and scored it 100/100 (green). The audit ran 55 deterministic rules across Security, Supply Chain, Maintenance, Transparency, and Community; it found 0 high-severity and 0 lower-severity findings. The full rule-by-rule trace and per-finding evidence are below. Free, methodology-open.
Findings & checks · 0 flagged
Every scanned point with the score it earned and what moved between them.
First recorded scan — no prior version to compare against.
The primary manifest — the file an agent reads to learn what this artifact does.
The complete paper-6 + paper-5 + paper-11 audit, packaged. Use this before claiming a probe is causal in any output the researcher will publish.
Always before publication. Never skip just because AUROC looks good.
Call the MCP tool `causality_protocol` with:
probe_id: loaded probecapture_id: stored activations + the original prompt (which must be one of the labeled examples)labels: binary labels for the captured rowsalpha_sweep (default [-200, -100, -50, 50, 100, 200]): α values to tryn_random_seeds (default 5): random-feature baseline seedsstructure_matched_capture_id + structure_matched_labels (Check 4a): a capture where theconfound (conversation structure, lexicon, punctuation, length, referent specificity) is balanced against the label. The SAME probe is re-evaluated on it. *Supply this whenever the claim is about what the feature means*** — without it the tool cannot return causal.
feature_name (Check 4b): the feature's max-activating-context name. Required before any causalclaim — high AUROC without naming is a non-result.
structural / lexical / punctuation confound, not the concept. Paper-11 shipped twice on a direction with AUROC 0.838 that collapsed to 0.08 once structure (and a stray em-dash) was balanced. So the signal must survive a confound-balanced set and the feature must be named.
| Verdict | What it means | What to recommend |
|---|---|---|
causal | Gap over baselines + Δrel + flips > 10%, survives the structure-matched control, and is named | Safe to publish as causal in this regime. |
weak-causal | Survives Check 4 + Δrel signal but flip rate < 10% | Paper-5 saturation-direction lever. Publish with the class label. |
confounded-structural | AUROC collapses on the structure-matched set | The direction reads structure/lexicon/punctuation, not the concept. Do NOT publish as causal. (paper-11 failure mode) |
epiphenomenal-softmax | Δrel ≈ 0 across all α | Uniform softmax-temp shifts. Do NOT publish as causal. |
epiphenomenal-template | Δrel ≈ 0 AND zero flips even at supra-norm α | Decision is in input tokens. Do NOT publish as causal. |
inconclusive-unverified | Stats + steering pass, but no structure-matched control and/or no name | NOT causal and NOT null. Run the missing control(s) before any claim. |
undetermined | AUROC gap below the N-adaptive floor | N too small / memorization. Need more data. |
Lead with the verdict and reason in two lines:
Verdict: {verdict} {reason}>
Real AUROC {real:.3f} vs random {rand:.3f} (gap +{gap:.3f}). Structure-matched {sm:.3f}. Δrel max {drm:.3f}, flip rate {fr:.0%} at α_max.
If the verdict is inconclusive-unverified, surface the exact missing control(s) from reason and offer to run them — do not present the result as causal. Then offer /publish only if the verdict is causal or weak-causal.
labels that don't match the captured rows. The tool will 400 you.causal on AUROC alone — that is exactly the paper-11 mistake. Without astructure-matched control and a feature name, the honest verdict is inconclusive-unverified.
epiphenomenal-template verdict. That class means the residualstream is not the causal locus regardless of magnitude.
confounded-structural verdict as fixable by more data — the signal is real but itis the wrong signal. Re-pose the contrast so structure/lexicon/punctuation no longer track the label.
~30 seconds. Free. No account. Every finding cites a rule and a line of evidence.