test-gap-analysis — independently scanned and version-tracked by SaferSkills.
SaferSkills independently audited test-gap-analysis (Agent Skill) and scored it 45/100 (orange). The audit ran 55 deterministic rules across Security, Supply Chain, Maintenance, Transparency, and Community; it found 7 high-severity and 0 lower-severity findings. The full rule-by-rule trace and per-finding evidence are below. Free, methodology-open.
Findings & checks · 7 flagged
A base64 string of 128+ characters appears in a documentation file. Encoded prompt injection hides the hostile instruction in base64 — invisible to keyword filters — and relies on the agent's ability to decode it at runtime. There is no normal authoring reason to embed a multi-hundred-byte base64 blob in skill docs.
*.sig, SIGNATURES) outside the documentation.A base64 string of 128+ characters appears in a documentation file. Encoded prompt injection hides the hostile instruction in base64 — invisible to keyword filters — and relies on the agent's ability to decode it at runtime. There is no normal authoring reason to embed a multi-hundred-byte base64 blob in skill docs.
*.sig, SIGNATURES) outside the documentation.A base64 string of 128+ characters appears in a documentation file. Encoded prompt injection hides the hostile instruction in base64 — invisible to keyword filters — and relies on the agent's ability to decode it at runtime. There is no normal authoring reason to embed a multi-hundred-byte base64 blob in skill docs.
*.sig, SIGNATURES) outside the documentation.A base64 string of 128+ characters appears in a documentation file. Encoded prompt injection hides the hostile instruction in base64 — invisible to keyword filters — and relies on the agent's ability to decode it at runtime. There is no normal authoring reason to embed a multi-hundred-byte base64 blob in skill docs.
*.sig, SIGNATURES) outside the documentation.A base64 string of 128+ characters appears in a documentation file. Encoded prompt injection hides the hostile instruction in base64 — invisible to keyword filters — and relies on the agent's ability to decode it at runtime. There is no normal authoring reason to embed a multi-hundred-byte base64 blob in skill docs.
*.sig, SIGNATURES) outside the documentation.A base64 string of 128+ characters appears in a documentation file. Encoded prompt injection hides the hostile instruction in base64 — invisible to keyword filters — and relies on the agent's ability to decode it at runtime. There is no normal authoring reason to embed a multi-hundred-byte base64 blob in skill docs.
*.sig, SIGNATURES) outside the documentation.A base64 string of 128+ characters appears in a documentation file. Encoded prompt injection hides the hostile instruction in base64 — invisible to keyword filters — and relies on the agent's ability to decode it at runtime. There is no normal authoring reason to embed a multi-hundred-byte base64 blob in skill docs.
*.sig, SIGNATURES) outside the documentation.Every scanned point with the score it earned and what moved between them.
First recorded scan — no prior version to compare against.
The primary manifest — the file an agent reads to learn what this artifact does.
Answer one question: which caller-visible production behaviors could change without an existing test failing? Mutation reasoning is a probe, not the goal. Inventory public outcomes first, then verify only credible gaps.
Before entering the decision flow, check the caller's mode. An explicit per-test-read-only request (including composition from grade-tests) uses only references/per-test-read-only.md. This mode is assessment context conveyed through the host's supported caller instructions, not a new skill-loader parameter or a mode-specific skill name. Read that reference and return its per-test assessment inline; do not enter the standalone workflow below. Loading this mode does not authorize a baseline run, mutation execution, file edits, suite discovery, or delegation. If the bundled reference is unavailable, allow at most one listing of the known references/ directory, then return N/A / unverified with the reason.
Requests without this mode retain the standalone analysis, verification, and test-addition paths below.
Discover production and test files from manifests and file types. After a narrow search misses, inspect the current directory broadly before asking for paths.
| Request | Action |
|---|---|
| One component or named risk | Inventory every high-risk public outcome in scope; do not edit production code unless verification was requested |
| General small-component review | Inventory distinct outcomes and report caller-visible gaps from source/assertion mapping |
| Explicit survivor verification | Inventory all requested outcomes; execute one representative observable candidate for each distinct high-risk outcome under verification, then classify it as Survived or Killed |
| Explicit exhaustive audit | Read references/mutation-catalog.md and classify all meaningful candidates |
| Add tests to an existing suite | Analyze first; add tests only for verified survivors or demonstrated no-coverage outcomes |
| Create a new suite | Stop and use code-testing |
When the request names a risk, turn it into a one-line public-outcome allowlist before reading code. An outcome is not in scope merely because the same method writes it. For money math, allow computed or returned amounts, rates, tier/boundary choice, percentage base/order, floors/caps, and rounding; exclude non-monetary state predicates (including derived booleans), identity, and formatting. Private code is in scope only to trace an allowed outcome.
Do not expand a focused request into a repository audit, plan artifact, or dashboard. Use source and tests directly for familiar frameworks. When discovery or assertion semantics are unclear, read the matching language file relative to the caller-provided or runtime-listed test-analysis-extensions catalog; do not invoke the reference-only helper or search installation directories. If absent, state the limitation and use the pinned APIs and rules here.
For a source-only advisory review, skip execution and label empirical survival unverified. A baseline is required when the user requests execution or survivor verification, or when closing gaps with tests. Run the narrowest existing test command once. Choose it from the project manifest; Microsoft.Testing.Platform executables may require dotnet run. Confirm tests executed: exit 0 with build-only output is not green. If that one attempt cannot run the suite, do not troubleshoot the runner or try alternate commands for an advisory review; continue statically and label all candidates unverified. Do not infer a project-configuration cause from missing output; name a cause only when the command reports it.
Missing runner output limits only claims of empirical mutation survival. It does not make source-proven facts tentative: a public outcome with no reaching test is still No coverage, and an exact expected value derived from the unmodified implementation is still actionable. State the baseline limitation once, then give the static source/assertion conclusion directly instead of hedging every row.
For an advisory review such as "would tests catch this?", stop execution after that baseline. Source-to-assertion mapping is sufficient evidence for No coverage and Candidate survivor (unverified). Trace or run the unmodified code once only when an original value is unclear. Apply mutations only for explicit verification, an exhaustive audit, or closing gaps with tests.
Any focused mutation budget limits execution, not discovery. Keep every distinct unasserted public outcome in the inventory.
For each public entry point, map:
nearest-valid guard boundaries, and default cases;
invalid-input acceptance, public state transition, or external side effect;
and error propagation as observed through the public caller.
Use public input/sequence -> expected outcome -> existing assertion -> gap. One asserted return field does not cover another. One allowed result does not cover its denial.
Money math: inventory the no-op path, every rate/tier and exact boundary, operation order, percentage base or composition, floor/cap, and rounding. Trace private helpers through the public result. A test asserting only a broad range does not pin any exact amount. For each actionable money row, derive one witness input and its exact original result through the complete call chain; do not recommend a generic "assert the exact amount" without supplying that amount.
Ordered guards and retries: inventory invalid below minimum | first valid | last allowed or retryable | first blocked | later blocked. For an upper guard such as value >= limit, use limit - 1, limit, and limit + 1; the last witness exposes narrowing to value == limit. Inventory every accepted and rejected error class. When type matching is polymorphic, include a representative derived accepted type that would expose exact-runtime-type narrowing. A test at the first blocked value does not protect the last allowed or later blocked value.
Authorization: enumerate each relevant identity/role, resource class, and action from the caller's view. Untested false, forbidden, and unchanged-role outcomes are first-class security gaps. Do not analyze variants of an allowed path while a denial outcome remains uninventoried. Check each public surface:
returned capability independently;
paths that must return denial;
including outcomes that must leave state unchanged.
Reserve execution for wholly untested public branches before another variant of a partially covered helper. If more than five high-risk behaviors are unasserted, execute the top 3-5 and keep the rest visible as No coverage or Candidate survivor (unverified).
Execution never replaces the ledger. Before mutating or answering, classify every required outcome, including each invalid input, guard boundary, classifier arm, action, and denial.
Completeness checkpoint: before selecting findings, explicitly account for every independent mode/flag, both zero and negative for a <= 0 guard, every accepted exception class, and a representative derived accepted exception when matching is polymorphic. For a removed guard, trace the fallthrough: if it still produces the same public exception type, it is equivalent unless finer exception metadata is an established contract.
First replay each exact mutation against every existing asserted input or sequence with all arguments fixed. Any changed return, exception, state, or side effect is Likely killed; a dedicated single-purpose test is unnecessary. Never compare the mutant on one input with the original on another.
For survivors, choose a witness before execution or reporting and state witness -> original observation -> mutant observation. Reuse it in the smallest test. Admit it only when the last two differ publicly after tracing the full call chain; otherwise choose a distinguishing witness or drop it.
Exclude:
removing an existing expression, condition, constant, return, or side effect;
still referenced;
ParamName metadata, or other semanticsnot established by the current contract, source intent, or tests;
exception, state, and side effects;
identically, even if the suite stays green;
example, changing result < floor ? floor : result to <= is equivalent at equality because both branches return floor;
meaningful branch, calculation, or side effect, unless the user names it;
impossible values, and duplicate syntax variants.
Missing direct assertions do not prove No coverage: first trace existing assertions through public callers and shared branches. Missing assertions make an observable candidate a survivor; they do not make an inert mutation meaningful.
Rank: (1) security denials, financial outcomes, errors, and state changes; (2) wholly unasserted public outcomes; (3) boundaries or exact values reached by weak assertions; (4) alternate variants of already-asserted behavior.
Finish the inventory before selecting mutations or a verdict. One killed attempt, exception type, or switch arm does not clear its siblings.
Choose the verdict from the completed inventory:
few validation or default-case variants remain;
partition is unprotected;
A handful of validation gaps does not make an otherwise broad suite Mixed unless validation is the named risk or the gaps threaten security, data, or other contract-critical behavior.
When the inventory meets the Strong criteria above, lead with Strong and name the protected boundaries and dual assertions before listing minor gaps. Do not open with Mixed, "only core paths", or a risk-heavy dashboard.
Stop when existing assertions kill the remaining candidates or no credible public survivor remains. Do not mutate every operator merely to fill a report or calculate a score.
| Result | Meaning |
|---|---|
| Likely killed | An existing assertion observes the changed outcome |
| Candidate survivor (unverified) | Observable change appears unasserted; not executed |
| Killed | Exact observable mutation executed and a relevant assertion failed |
| Survived | Exact observable mutation executed and tests stayed green |
| No coverage | No test reaches the public outcome; report the missing branch without inventing a survivor |
| Equivalent | No public observation changes; omit from findings |
Outside explicit verification, an exhaustive audit, or a requested test addition, execute no mutations. Do not mutate to confirm obvious no coverage. For explicit verification, execute one representative candidate per distinct high-risk outcome in scope; do not stop after the first one or two while another guard, action branch, error class, or denial remains unclassified. Omit equivalent syntax variants.
Enter this phase only for explicit verification, an exhaustive audit, or a requested test addition.
expression.
Survived; a failure at a relevant assertion means Killed, for that edit only. Build, discovery, infrastructure, or unrelated failures leave the candidate unverified, not Killed.
suite missed the edit, not that the edit changes behavior; drop inert or unobservable mutants.
Never leave mutations in the workspace. Before reporting, reconcile every unasserted high-risk outcome as Survived, Candidate survivor (unverified), No coverage, or omitted Equivalent. Stop when no credible public gap remains; do not fill a report with internal details or calculate a score unless the user requested an exhaustive audit.
No coverage behavior.
for alternate variants of an already-covered behavior.
every verified survivor to an added test and every added test back to a verified survivor; a passing final suite alone does not prove completeness.
per syntax change.
the source and run the narrow suite cleanly.
that exact command after the tests are added and cite its successful result. Hand-created substitute mutations, a broad green suite, or a test-count increase do not replace the supplied oracle. Once every requested survivor maps to a focused test and the canonical verifier passes, stop; extra tests are not an advantage.
compare each protected file byte-for-byte with its pre-edit snapshot and report that evidence. Before adding a test, prove its witness differs from every existing case on the relevant branch, boundary, or rounded result so a nominally new test does not duplicate existing coverage.
Scale the response to the request.
For focused or small analysis, return:
protected boundaries, guards, or paired observations that justify the verdict.
(unverified), or No coverage** outcome. Before adding a row, apply the outcome allowlist when the request names a risk, then apply the observable-candidate rules; omit any candidate that fails either filter. Include every high-risk outcome, use one row per distinct public outcome, and consolidate only related low-risk variants:
| Risk | Public outcome | Change | Result/evidence | Smallest test |
|---|
Every gap needs a distinguishing witness and a concrete smallest test. An error-path gap must name an invalid input and the expected error/result.
important killed behavior.
generated, trivial, or unrelated code intentionally skipped.
Do not repeat the table in prose or report discarded mutants, tool chronology, or in-flight reasoning.
For an exhaustive audit, separate executed Killed / Survived counts from static Likely killed / Candidate survivor (unverified) classifications and No coverage / Equivalent inventory counts. Never include static reasoning in an empirical killed/total ratio. Group findings by risk.
For test additions, name the tests added, the verified mutations they kill, and the successful final command.
mutation.
does not prove siblings: allow does not prove deny; read does not prove write; null does not prove empty or whitespace when those inputs have different caller-visible outcomes. A kill clears only the edit and path that ran.
? propagation versus panicis observable behavior; in C#, exception type and whether an input guard accepts or rejects a value are observable behavior.
implementation or an existing exact assertion. If it cannot be checked, state the behavioral relation without inventing a number.
protects.
Candidate survivor (unverified)
is explicitly accounted for
with an ad-hoc proxy
~30 seconds. Free. No account. Every finding cites a rule and a line of evidence.