grade-tests — independently scanned and version-tracked by SaferSkills.
SaferSkills independently audited grade-tests (Agent Skill) and scored it 45/100 (orange). The audit ran 55 deterministic rules across Security, Supply Chain, Maintenance, Transparency, and Community; it found 7 high-severity and 0 lower-severity findings. The full rule-by-rule trace and per-finding evidence are below. Free, methodology-open.
Findings & checks · 7 flagged
A base64 string of 128+ characters appears in a documentation file. Encoded prompt injection hides the hostile instruction in base64 — invisible to keyword filters — and relies on the agent's ability to decode it at runtime. There is no normal authoring reason to embed a multi-hundred-byte base64 blob in skill docs.
*.sig, SIGNATURES) outside the documentation.A base64 string of 128+ characters appears in a documentation file. Encoded prompt injection hides the hostile instruction in base64 — invisible to keyword filters — and relies on the agent's ability to decode it at runtime. There is no normal authoring reason to embed a multi-hundred-byte base64 blob in skill docs.
*.sig, SIGNATURES) outside the documentation.A base64 string of 128+ characters appears in a documentation file. Encoded prompt injection hides the hostile instruction in base64 — invisible to keyword filters — and relies on the agent's ability to decode it at runtime. There is no normal authoring reason to embed a multi-hundred-byte base64 blob in skill docs.
*.sig, SIGNATURES) outside the documentation.A base64 string of 128+ characters appears in a documentation file. Encoded prompt injection hides the hostile instruction in base64 — invisible to keyword filters — and relies on the agent's ability to decode it at runtime. There is no normal authoring reason to embed a multi-hundred-byte base64 blob in skill docs.
*.sig, SIGNATURES) outside the documentation.A base64 string of 128+ characters appears in a documentation file. Encoded prompt injection hides the hostile instruction in base64 — invisible to keyword filters — and relies on the agent's ability to decode it at runtime. There is no normal authoring reason to embed a multi-hundred-byte base64 blob in skill docs.
*.sig, SIGNATURES) outside the documentation.A base64 string of 128+ characters appears in a documentation file. Encoded prompt injection hides the hostile instruction in base64 — invisible to keyword filters — and relies on the agent's ability to decode it at runtime. There is no normal authoring reason to embed a multi-hundred-byte base64 blob in skill docs.
*.sig, SIGNATURES) outside the documentation.A base64 string of 128+ characters appears in a documentation file. Encoded prompt injection hides the hostile instruction in base64 — invisible to keyword filters — and relies on the agent's ability to decode it at runtime. There is no normal authoring reason to embed a multi-hundred-byte base64 blob in skill docs.
*.sig, SIGNATURES) outside the documentation.Every scanned point with the score it earned and what moved between them.
First recorded scan — no prior version to compare against.
The primary manifest — the file an agent reads to learn what this artifact does.
Assess a curated list of test methods and produce a compact, PR-comment-friendly report. The primary result is one of Pass, Failed, Uncertain, or Not applicable; an A-F quality grade remains secondary diagnostic information. The skill does not discover tests on its own — the caller (typically a PR automation workflow or a human reviewer holding a specific list) provides the tests or a bounded diff to assess.
After Step 0 admits a bounded scope, enforce these grading invariants:
test-gap-analysis by name once beforescoring**, in per-test-read-only caller context. Read its owned composition reference; do not compute mutation evidence from this grading rubric or run its standalone workflow. Report N/A / unverified only when the dependency, reference, or required context is actually unavailable.
Candidate survivor (unverified), even when explained in prose.
penalty. Do not use sibling tests to alter the individual assessment.
test may have no actionable change.
Language-specific guidance: Read the caller-provided matching extension file, or the runtime-listed test-analysis-extensions catalog and its matching language file. The reference-only helper is not an invocable skill. If its files are absent, use the embedded rubric and pinned framework APIs; mark an essential unresolved framework construct Uncertain instead of guessing or searching installation directories.PR reviewers need a simple answer to does this test need follow-up? The four-state result provides that decision; the existing A-F rubric explains its quality and severity.
modified in a pull request.
or a diff hunk) and wants per-test follow-up decisions rather than a suite report.
follow-up improvements, with quality grades for resolved tests.
test-anti-patterns (pragmatic) or test-smell-detection (formal) and let the test-engineer agent orchestrate its internal quality specialist.
test-engineer(any language) or writing-mstest-tests (MSTest specifically).
coverage-analysis or crap-score (.NET only).
appropriate editing skill.
in the workspace; ask the caller for an explicit list or scope.
| Input | Required | Description |
|---|---|---|
| Test methods | Yes | A scope to grade. Provide one of: (a) an explicit list of test method names (fully-qualified, e.g. Namespace.ClassName.TestMethodName); (b) one or more file paths plus an explicit instruction to grade every test declared in those files; or (c) a diff hunk / PR identifier whose changed tests should be graded. File paths are recommended but optional when method names are unambiguous in the workspace. Ambiguous requests like "grade my tests" with no scope are rejected up-front (see Step 0); this skill is for curated input and does not auto-grade an entire workspace. |
| Test bodies / spans | Recommended | The exact source lines for each test method. If omitted, read them from the listed files. |
| Production code | No | The code under test, for judging whether assertions cover the claimed behavior. When unavailable, mark the mutation assessment N/A / unverified rather than guessing or deducting. |
| Language reference | No | A host-supplied path to the matching bundled test-analysis-extensions file. Read it directly instead of invoking its reference-only loader; do not substitute unverified framework guidance. |
| Diff context | No | When grading PR changes, the unified diff for each test method helps focus on what actually changed. |
Before doing anything else, check that the caller provided one of:
declared in those files (e.g., "grade every test in OrderTests.cs"), or
If the request is ambiguous (e.g., "Grade my tests", "Are these tests any good?" with no scope, "Review the test suite"), do not load extensions, do not read files, and do not grade anything. Reply with a short message asking the caller to provide an explicit list / file(s) / diff, and optionally point them at the test-engineer agent or test-anti-patterns skill for full-suite analysis. Stop there.
If a valid bounded scope resolves to zero eligible tests, return Not applicable with a short explanation and no invented rows.
Identify the target codebase's language and test framework from the file extensions and the test method markers in the provided list. Read the matching caller-provided extension file directly; otherwise read the matching extensions/ file relative to the runtime-listed test-analysis-extensions catalog (dotnet.md, python.md, typescript.md, go.md, etc.). Use the catalog for other languages, not a skill invocation. If the input contains tests from multiple languages, load each relevant extension and grade each test using its language's conventions.
For each entry in the input list:
fully-qualified name. Capture the full method body, including attributes / decorators / fixtures and any helper code that the test calls.
Uncertain — method not found with no quality grade and continue. Never invent a body to grade. A missing requested method requires human review; it is not the same as a valid scope containing no tests.
Composition checkpoint: for resolved tests with available production context, load test-gap-analysis now, once for the batch, with per-test-read-only assessment context. Complete its owned reference assessment before Step 3. Do not skip this load just because a body-level weakness already seems obvious; a locally invented mutation explanation is not composition.
Keep grading read-only: no build/test runs, mutation execution, file edits, tool installation, broad suite discovery, or agent delegation. Resolve only the supplied tests, their relevant fixtures/helpers, and the production call chain needed for their claims.
Use the inline test-gap-analysis assessment from Step 2's checkpoint; do not load it a second time. Supply each test's identifier/body, relevant setup/helpers, claimed behavior, assertion semantics, and available source. Its composition dispatch loads the owned read-only reference rather than its standalone baseline/verification workflow. Consume its per-test evidence; do not duplicate its mutation catalog here or invoke an audit/generation agent. Convey mode and inputs as assessment context using the host's supported caller instructions. If the loader accepts only a skill name, load test-gap-analysis by name only; do not invent tool arguments or a mode-specific skill name.
If the skill/reference or production context is unavailable, record Pseudo-mutation: N/A / unverified — <reason> and continue normal body-level grading. This is not a grade deduction or, by itself, an Uncertain result. Do not search installation directories or substitute a mutation runner.
Assess only what each test claims: do not borrow another test's assertions, or demand unrelated branches, outputs, or scenarios. An observable survivor can support an existing Assertion strength category when it proves that the test does not verify its claimed outcome; do not introduce mutation points, weights, ceilings, or an automatic survivor penalty. Apply the existing rubric normally, including weaknesses it classifies in both Assertion and Anti-pattern dimensions; do not add another deduction for the same mutation evidence.
Start every test at grade A (score band 90–100), then apply deductions strictly for observable issues in the captured body. Do not deduct for hypothetical concerns (e.g., "could have more negative assertions") unless the production code clearly demands them and the production code is available.
When production code is unavailable, grade observable issues in the test body normally, but do not infer missing behaviors or deduct for them. State Production-dependent behavior coverage: Unverified once in the summary so the reader can distinguish test-body findings from claims that require source code.
#### Three sub-dimensions
Compute three sub-grades (each A–F) that together drive the overall grade.
##### A. Assertion strength
Read the loaded language extension's assertion API list and classify every assertion in the test body. Score from highest to lowest:
| Sub-grade | Pattern |
|---|---|
| A | Pins a richer contract through structural equality or nonredundant related observables (such as returned values plus required state/interaction checks), or qualifies for the exception/error calibration below. Grade predicate depth, not assertion count. Mock-call verifications and bare language assertions count normally. |
| B | One clear meaningful assertion that verifies the behavior under test. |
| C | Only trivial assertions (single IsNotNull / toBeDefined / assert x is not None), or assertions that leave a meaningful part of the test's claimed result unchecked. A focused single-field claim does not require unrelated fields. |
| D | One self-referential / tautological assertion (Assert.AreEqual(x, x), assert dto.name == dto.name, round-trip identity without a non-trivial input), or broad exception assertions (Assert.ThrowsException<Exception>). |
| F | No assertions at all; all assertions are always-true literals (Assert.IsTrue(true), assert True, expect(true).toBe(true)) — these verify nothing and are equivalent to having no assertions; or all assertions are silently un-awaited (e.g., expect(promise).resolves.toBe(x) without await/return, async TUnit/xUnit Assert.ThrowsAsync without await, pytest-asyncio with un-awaited coroutine). |
Exception and error-path tests (Assert.ThrowsException<T>, constrained pytest.raises, expect(fn).toThrow, assertThrows, #[should_panic], Should -Throw, EXPECT_THROW, or Go code that verifies an expected non-nil error) are complete on their own. Give Assertion strength A when the test checks the exact promised error condition for its stated scope. Do not deduct for having only that assertion, and do not require an error-message assertion unless the message is part of the documented contract. A Go happy-path test that only checks err == nil while discarding a meaningful returned value is still C because it does not verify the successful result.
##### B. Structure & focus
| Sub-grade | Pattern |
|---|---|
| A | Clear Arrange-Act-Assert (or Given-When-Then) separation. Single behavior under test. Body under ~30 lines. Setup uses framework conventions. |
| B | One mild structural issue (slightly long body, missing blank lines between phases) but intent is clear. |
| C | Multiple behaviors mixed in one test, or AAA phases interleaved enough to slow comprehension. |
| D | Conditional logic in the test (if/switch driving assertions) — except for idiomatic Go/Rust table-driven sub-test loops; or test relies on previous test state (ordering dependency). |
| F | Test exceeds ~60 lines and verifies multiple unrelated behaviors; or shares mutable state with other tests through statics/globals without reset. |
##### C. Anti-pattern hygiene
Scan against the catalog below. The Anti-pattern sub-grade is computed in two passes and combined deterministically:
maximum sub-grade (F, D, or C as labeled). Take the worst ceiling across all matched Critical/High findings — these do not accumulate (a single F finding caps the sub-grade at F regardless of how many other Critical/High findings are present).
finding deduct one sub-grade level (A→B, B→C, C→D, D→F). These do accumulate across findings.
The final Anti-pattern sub-grade is the worse of the two passes (i.e., min(hard_ceiling, A − medium_count)). Low findings never affect the grade — mention them in the note only.
Examples (Critical/High and Medium counts → Anti-pattern sub-grade):
min(C, A − 2 = C) = C; a third Medium tips to D)Critical (drop straight to F or D)
try { … } catch { } (.NET), bare except: pass(Python), try { … } catch (e) {} (JS/TS/Java), defer recover() without re-panic (Go), rescue StandardError with no assertion (Ruby), empty catch (Kotlin/Swift) → F
Assert.Fail(ex.Message) instead ofAssert.ThrowsException) → D
Assert.IsTrue(true), assert True,expect(true).toBe(true)) → F (verifies nothing; also drives Assertion sub-grade to F)
(Assert.AreEqual(x, x), assert dto.name == dto.name) → D
High (drop one or two sub-grades)
Thread.Sleep, Task.Delay,time.sleep, setTimeout-based wait, Thread.sleep, time.Sleep, sleep, std::thread::sleep, Start-Sleep, std::this_thread::sleep_for (in a unit test) → D
(DateTime.Now, datetime.now(), Date.now(), System.currentTimeMillis(), time.Now(), Time.now, Instant::now(), Get-Date, system_clock::now) → D
C:\…, /tmp/…, network hosts) → DAssert.ThrowsException<Exception>,pytest.raises(Exception), expect(fn).toThrow(Error) without matcher, #[should_panic] without expected = "…", Should -Throw without -ExpectedMessage, EXPECT_ANY_THROW) → C
call sequences instead of outcomes → C
internal types to access state → C
Medium (drop one sub-grade)
Test1, TestMethod, test, single-word name that saysnothing about scenario or expected outcome (judge against the language extension's convention) → drop one sub-grade
42, "foo", 0x1234 in arrange/assertwithout naming or comment → drop one sub-grade
Low (note only, no deduction)
Console.WriteLine,print, console.log, System.out.println, fmt.Println, puts, dbg!, Write-Host, std::cout); inconsistent naming versus siblings; leftover TODO comments. Mention in the note column but do not deduct.
#### Combining sub-grades
Convert sub-grades to numeric points: A=4, B=3, C=2, D=1, F=0.
0.45 × Assertion + 0.30 × Anti-pattern + 0.25 × Structure
is F, the overall grade is F; if the worst sub-grade is D, the overall grade is at most D; and so on. A test that fails on any one dimension cannot earn a higher overall grade than that dimension.
Apply the cap after the weighted mapping, not instead of it. For example, Assertion B, Anti-pattern A, Structure A gives 3.55: the provisional weighted grade is A, but the final grade is B (80-89). A focused scalar equality can therefore be B / Pass with no actionable finding; do not invent a weakness or missing sibling behavior to justify that grade.
Report the letter grade and the score band (not a single 0–100 number). False precision invites bikeshedding; bands keep the conversation focused on the rubric.
The grade summarizes strength; the result says whether follow-up exists. An actionable improvement is an evidence-backed change to the test, setup, or fixtures. Assign exactly one:
overall result with no rows.
Do not derive status from grade: a complete focused test can be B / Pass, while debug output can make an otherwise excellent test A / Failed. Use Uncertain for an unresolved body, unsupported construct, or essential missing contract—not merely absent production code. A definite finding wins over uncertainty.
Use one sentence (target ≤ 120 characters) for the most important reason: No issues found., Only checks IsNotNull; receipt contents are unverified., or Method body could not be resolved; human review is required. Do not invent a weakness to justify a grade or Failed result.
Keep the action in a separate How to improve field. For each Failed test, name the smallest useful input, assertion, or fixture change and its expected outcome, grounded in the body, source, or an explicit contract. For example, Replace self-comparison with Assert.AreEqual(60m, account.Balance)., not Improve assertions; Remove Console.WriteLine after Deposit(25m)., not Clean up. Prioritize the highest-impact distinct finding, and include other actionable findings only when they require a different change.
For a behavioral gap, use the distinguishing witness and original/mutant observations from the shared assessment; check the expected result against the unmodified source. If essential context is missing, name the evidence needed instead of inventing an expected value. Pass gets None; Uncertain gets a concrete evidence-resolution step, not a speculative test rewrite. A rubric-only deduction is not proof of a behavioral gap or an actionable improvement: a focused B / Pass may need no change. A / Failed still needs its concrete action, such as removing debug output.
Produce two sections.
#### 1. Summary
Begin with **Result: <Pass|Failed|Uncertain|Not applicable>**, then give result counts and the highest-priority action. Aggregate using Failed → Uncertain → Pass → Not applicable. For Not applicable, explain the empty scope and omit the table.
#### 2. Per-test table
| Test | Result | Quality | Notes | How to improve |
|------|--------|---------|-------|----------------|
| `Namespace.ClassName.Test_Method_Condition_Expected` | Pass | B (80–89) | One complete value assertion. | None |
| `Namespace.ClassName.Withdraw_SufficientFunds` | Failed | D (60–69) | Balance is compared with itself. | Replace self-comparison with `Assert.AreEqual(60m, account.Balance)` after withdrawing 40m from 100m. |
| `Namespace.ClassName.Test_Missing` | Uncertain | — | Method body could not be resolved; human review is required. | Supply the method body and its referenced fixture. |Keep these two report sections and the original Test/Result/Quality/Notes fields. When mutation evidence explains a finding or the caller requests detail, append a compact per-test Pseudo-mutation evidence block inside the per-test section: change, witness, original/mutant observations, relevant assertion, and classification. Static results are Likely killed (inferred) or Candidate survivor (unverified), never executed Killed/Survived or empirical killed/total counts. State missing-context N/A / unverified once per shared limitation. Do not repeat the improvement table in prose.
Caps and ordering:
Uncertain tests, then a sample of Pass tests. Wrap overflow in a collapsed <details> block.
path and method name for determinism.
(new) or(modified) marker.
If multiple languages are present, produce one table per language and prefix each section with the language name and framework.
Uncertain — method not found).
captured body — no speculative deductions.
Pass rows have no invented weakness, even when the quality grade is B.
was not penalized, equivalents were excluded, and static labels/counts were not presented as executed evidence.
duplicate deduction or any change to scoring weights and ceilings.
is trivial (a null check before a meaningful assertion is not trivial).
assertions of the appropriate category.
Assert.IsTrue(result.IsValid))are not classified as always-true; only literal true/false constants are.
equality assertions.
pytest bare assert, Go if got != want { t.Errorf(...) }, JS/TS expect(mock).toHaveBeenCalledWith(...).
resolves/rejects/ThrowsAsync,pytest-asyncio without await) drop the Assertion sub-grade to F.
of the table.
| Pitfall | Solution |
|---|---|
| Grading every test in the workspace when no list is provided | Ask the caller for the explicit list; this skill is for curated input. |
| Inflating deductions to justify the grade | Start at A; deduct only for observable issues. |
| Penalizing exception tests for low assertion count | Exception assertions are complete on their own. |
Downgrading a focused Go error-path test because it checks only err != nil | Expected-error existence is the observable contract for that scope; keep it at A unless the production contract requires a specific error identity or message. |
Treating IsNotNull before a value assertion as trivial | Only flag when the null check is the only assertion. |
| Treating any Boolean assertion as effectively assertion-free | Only always-true literals (Assert.IsTrue(true), assert True) are; meaningful Assert.IsTrue(result.IsValid) is a real assertion. |
| Flagging Go/Rust table-driven loops as conditional logic | They are idiomatic; do not deduct. |
Treating pytest bare assert or Go if got != want { t.Error… } as missing-framework | Both are canonical; count in the correct assertion category. |
| Penalizing tests when production code is unavailable | Mark concerns about uncovered behaviors as Unverified and do not deduct. |
| Using a fake-precise score (e.g., 87/100) | Use the score band only — 90–100, 80–89, 70–79, 60–69, 0–59. |
| Spilling a 500-row table into a PR comment | Apply the row cap from Step 6; collapse extras into <details>. |
| Re-reporting an existing finding three times under different categories | Pick the most fitting category and report once. |
| Giving a weak test credit for a sibling's assertions | Use only the current test and helpers/fixtures it executes. |
| Turning pseudo-mutation composition into a suite audit | Pass explicit per-test-read-only mode; no runs, edits, broad discovery, or agent recursion. |
| Inventing weaknesses for A-grade tests to make the note "balanced" | If a test is clean, the note may simply read No issues found. |
| Mapping status from grade or comments | Fail only for actionable improvements; a B can Pass and an A can Fail. |
| Confusing Uncertain and Not applicable | Evidence gaps are Uncertain; a valid empty scope is Not applicable. |
~30 seconds. Free. No account. Every finding cites a rule and a line of evidence.