test-smell-detection — independently scanned and version-tracked by SaferSkills.
SaferSkills independently audited test-smell-detection (Agent Skill) and scored it 45/100 (orange). The audit ran 55 deterministic rules across Security, Supply Chain, Maintenance, Transparency, and Community; it found 7 high-severity and 0 lower-severity findings. The full rule-by-rule trace and per-finding evidence are below. Free, methodology-open.
Findings & checks · 7 flagged
A base64 string of 128+ characters appears in a documentation file. Encoded prompt injection hides the hostile instruction in base64 — invisible to keyword filters — and relies on the agent's ability to decode it at runtime. There is no normal authoring reason to embed a multi-hundred-byte base64 blob in skill docs.
*.sig, SIGNATURES) outside the documentation.A base64 string of 128+ characters appears in a documentation file. Encoded prompt injection hides the hostile instruction in base64 — invisible to keyword filters — and relies on the agent's ability to decode it at runtime. There is no normal authoring reason to embed a multi-hundred-byte base64 blob in skill docs.
*.sig, SIGNATURES) outside the documentation.A base64 string of 128+ characters appears in a documentation file. Encoded prompt injection hides the hostile instruction in base64 — invisible to keyword filters — and relies on the agent's ability to decode it at runtime. There is no normal authoring reason to embed a multi-hundred-byte base64 blob in skill docs.
*.sig, SIGNATURES) outside the documentation.A base64 string of 128+ characters appears in a documentation file. Encoded prompt injection hides the hostile instruction in base64 — invisible to keyword filters — and relies on the agent's ability to decode it at runtime. There is no normal authoring reason to embed a multi-hundred-byte base64 blob in skill docs.
*.sig, SIGNATURES) outside the documentation.A base64 string of 128+ characters appears in a documentation file. Encoded prompt injection hides the hostile instruction in base64 — invisible to keyword filters — and relies on the agent's ability to decode it at runtime. There is no normal authoring reason to embed a multi-hundred-byte base64 blob in skill docs.
*.sig, SIGNATURES) outside the documentation.A base64 string of 128+ characters appears in a documentation file. Encoded prompt injection hides the hostile instruction in base64 — invisible to keyword filters — and relies on the agent's ability to decode it at runtime. There is no normal authoring reason to embed a multi-hundred-byte base64 blob in skill docs.
*.sig, SIGNATURES) outside the documentation.A base64 string of 128+ characters appears in a documentation file. Encoded prompt injection hides the hostile instruction in base64 — invisible to keyword filters — and relies on the agent's ability to decode it at runtime. There is no normal authoring reason to embed a multi-hundred-byte base64 blob in skill docs.
*.sig, SIGNATURES) outside the documentation.Every scanned point with the score it earned and what moved between them.
First recorded scan — no prior version to compare against.
The primary manifest — the file an agent reads to learn what this artifact does.
Audit test code with the academic taxonomy, code evidence, calibrated framework idioms, and fixes native to the codebase.
for code; never claim a file is missing until that search finds no relevant test.
the caller-provided or runtime-listed test-analysis-extensions catalog, not a skill invocation. If absent, use the framework and rules here and report the missing reference; do not search installation directories.
requests all 19 smells, asks for citations, or the code may contain a smell outside the high-signal set below. Do not load it for a narrow question that this file answers.
and integration markers. This is the first action even when no path is named.
calibrate, then assign severity. Report proven non-catalog test-validity defects separately; do not relabel them as smells.
maintenance cost.
.NET terminology or APIs in another ecosystem.
| Evidence | Academic finding | Do | Never |
|---|---|---|---|
Assertion behavior changes behind if, switch, or branching loops | Conditional Test Logic | Split cases or parameterize them | Flag table-driven or parametrized tests merely because a runner loop exists |
| A test relies on an undeclared file, network service, environment value, or database | Mystery Guest or Resource Optimism | Make the dependency explicit and hermetic; distinguish the two using the full catalog | Condemn an integration test merely for exercising its declared real resource |
| Fixed wall-clock sleep waits for an outcome | Sleepy Test | Await or poll the condition with a timeout | Downgrade it only because the test is an integration test |
| Executable test has no assertion, expected-exception marker, or mock verification | Unknown Test | Assert the observable outcome | Call an empty body Unknown Test; the formal name is Empty Test |
| Async assertion/coroutine is created but not awaited or returned | Critical non-catalog false-pass defect | Report it separately and show the required await/return | Force it into Unknown Test; the assertion statement exists |
| One test exercises many unrelated production behaviors | Eager Test | Separate behavior-focused tests | Flag a deliberate end-to-end workflow without considering its scope |
| Expected numeric literal has no local meaning | Magic Number Test | Name the domain value or derive it from setup | Flag count == 3 immediately after adding three items |
Assertion depends on ToString, repr, description, or display formatting that is not the contract | Sensitive Equality | Assert stable fields or use a structural matcher | Flag a test whose explicit contract is the formatted string |
| Test manually manages expected exception flow | Exception Handling | Use the framework's exception assertion and check meaningful details | Claim a capture-and-assert test verifies nothing |
| Shared setup creates state irrelevant to the tests that receive it | General Fixture | Remove unused state or narrow the fixture; rank cheap state low | Condemn relevant shared setup merely because it is shared |
| Test is disabled or skipped | Ignored Test | Report every skip, but rank a tracked, reasoned skip below an unexplained one | Clear a skip because its reason is good, or give both the same urgency |
Apply these before assigning a finding:
assert, PesterShould -Invoke, and expected-exception constructs are assertions.
Test or another smell without separate evidence.
missing messages alone are not a smell.
boundaries, or state; require redundant equivalent paths.
is neither General Fixture nor Test Code Duplication by itself.
production context or requirements make them display-only; interpolation alone is not Sensitive Equality.
setup quantities whose role is locally obvious and irrelevant to the asserted behavior.
.each, RSpec data tables, Pester -ForEach, and Catch2 SECTION/GENERATE are not Conditional Test Logic by themselves.
if err != nil { t.Fatal(...) } is idiomatic assertion flow, notException Handling.
flows, but not fixed sleeps or assertion-free execution.
before judging branch coverage. If a test branches on a subtype or state that the real helper can never produce, report the unreachable assertion path and explain the resulting false confidence. Do not stop at assigning the generic Conditional Test Logic label.
Hermetic creation and cleanup reduce its severity; they do not change its taxonomy.
it from production behavior or requirements before clearing Sensitive Equality.
fixture relationship that proves it.
alone. Without production code or an explicit contract proving that the test is supposed to invoke another component, a name/body mismatch is at most an unranked observation, not evidence that the test silently passes broken production behavior.
SECTION and GENERATE are runner-controlled case expansion, andREQUIRE is a real assertion. When those are the only suspicious constructs, the academic-smell verdict is clean. Do not reverse that verdict because the test could have broader behavioral coverage.
manufacture findings to fill a report.
await for a void or otherwise non-awaitable API. If productionwork is synchronous, remove the sleep and assert immediately.
Severity follows demonstrated risk, not a fixed label copied from the catalog:
nondeterministic failures, or hides unexecuted assertion paths.
cheap fixture.
State the reason for the assigned severity. Downgrade or omit a finding when the surrounding test type makes the pattern intentional.
Scale the response to the input:
formal smell, evidence, risk, and fix.
findings across dashboards, prose, and plans.
clean input with no production contract, do not assign severity to optional coverage observations or let them overturn the clean verdict.
Every reported smell must have a formal taxonomy name, precise location, evidence from the code, practical risk, and a concrete framework-correct fix.
remains Mystery Guest; rationale and hermetic cleanup change severity only.
findings from naming or absent production context.
performed.
~30 seconds. Free. No account. Every finding cites a rule and a line of evidence.