testweaver

Generate mutation tests for one Go package per invocation.

Inputs

• $1 — Go import path of the target package (required, e.g. github.com/acme/repo/pkg/foo).
• $2 — max mutations to attempt this run (optional, default 1).

If $1 is missing, ask the user and stop.

Reference files (load with Read when needed)

Do not preload all three at once — only what the current step needs.

Hard rules (NEVER violate)

1. Hash round-trip is mandatory. Before mutating a source file, capture git hash-object <file>. After git checkout -- <file>, recompute and compare. Mismatch → ABORT the entire run. Working tree is corrupt; do not commit anything.
2. If the operator at the target offset does not match any transform for the mutation type, ABORT that mutation — mark it skipped, do not write a partial edit. Never guess which byte to change.
3. NEVER mark a mutation `tested` without a successful hash round-trip AND a confirmed new-test failure on mutated code AND a passing new-test on unmutated code.
4. NEVER commit unless `git status` shows only the generated `_test.go` files and any `//gremlins:skip` annotations. No source-file edits should ever be staged.
5. Update `state.toml` before each phase transition (mark attempted before doing work). On error, the state file must still reflect what was tried.

Flow

1. Preflight

command -v gremlins >/dev/null || { echo "gremlins not installed"; exit 1; }
command -v gh       >/dev/null || { echo "gh not installed"; exit 1; }
command -v jq       >/dev/null || { echo "jq not installed"; exit 1; }
REPO_ROOT=$(git rev-parse --show-toplevel)
cd "$REPO_ROOT"
test -f .testweaver/allowlist.toml || echo "(no allowlist — proceeding with no exclusions)"
mkdir -p .testweaver
test -f .testweaver/state.toml || printf '[cursor]\ncurrent_package = ""\n\n[packages]\n' > .testweaver/state.toml

Verify working tree is clean: git status --porcelain must be empty. If not, stop and ask the user.

2. Allowlist filter

Read .testweaver/allowlist.toml. If $1 appears in exclude_packages, report "package excluded" and stop.

3. Discover mutations

Resolve the package directory and run gremlins:

PKG_DIR=$(go list -f '{{.Dir}}' "$1")
REL_DIR="./${PKG_DIR#$REPO_ROOT/}"
TMP=$(mktemp /tmp/gremlins-XXXX.json)
gremlins unleash --output "$TMP" "$REL_DIR" || true   # non-zero exit is OK

Parse $TMP with jq. Build a list of mutations where status == "LIVED":

jq -r '.files[] | .file_name as $f | .mutations[] | select(.status=="LIVED") | "\(.type)|\($f)|\(.line)|\(.column)"' "$TMP"

For each entry:

• Compute repo-relative file path: $REL_DIR/<file_name> normalized.
• Compute mutation ID: <type>:<repo-relative-file>:<line>:<column> (see references/state-schema.md).
• Apply allowlist exclude_files (glob) and exclude_mutation_types.
• Drop any ID already present in state.toml under this package's attempted, tested, or skipped lists.

If zero remain, report "no new surviving mutations" and stop.

4. Sort and cap

Process at most $2 (default 1) mutations. Pick by ascending (file, line, column) for deterministic behavior.

5. Per-mutation loop

For each selected mutation, run steps 6–10 below. Stop the loop after the configured count of successful catches (not attempts).

6. Mark attempted

Append the mutation ID to packages.<pkg>.attempted in state.toml, using the atomic write pattern from references/state-schema.md.

7. Generate & validate

Use up to 3 generation attempts per mutation. On each attempt:

a. Gather context. Read:

• The source file in full (the file containing the mutation).
• The companion test file <basename>_test.go (same directory as the source file, where <basename> is the source file with .go stripped), if it exists. Also note any other *_test.go files in the package so the package clause and helpers can be matched.
• go.mod for module path and Go version.
• The 10 lines around (line, column) of the mutation.

b. Decide test target. Determine which exported function (or behavior) the mutation lives inside. If it's an unexported helper called by exported code, target the exported caller. If unreachable from any exported entry point, mark skipped with reason "unreachable from exported API" and move on.

c. Write the test. Pick a unique name Test<Func>_<MutationType>_<Line> (e.g. TestParse_CONDITIONALS_BOUNDARY_42). Target file selection:

1. If <basename>_test.go exists in the same directory as the source file, append the new test to it using Edit.
2. Otherwise, create a new <basename>_test.go in the same directory. Never use a _testweaver suffix — the conventional <basename>_test.go name is required even when no test file exists yet.

Use the same package clause as other *_test.go files in the package (preferring the internal package over _test package unless existing tests use _test). Do not append to an unrelated existing test file (e.g. helpers_test.go) — only the file matching <basename>_test.go.

d. Baseline test run (only on first attempt for this mutation — cache the result):

go test -v -json -count=1 "$1" > /tmp/baseline.json 2>&1 || true

Extract failing tests + panic flag:

jq -r 'select(.Action=="fail" and .Test!=null) | .Test' /tmp/baseline.json | sort -u > /tmp/baseline.failed
grep -q '^panic:' <(jq -r 'select(.Output!=null) | .Output' /tmp/baseline.json) && echo PANIC > /tmp/baseline.panic || true

If /tmp/baseline.panic is non-empty → outcome baseline-broken, abort this mutation (don't retry — it's a repo-level issue). Mark skipped.

e. Compile check.

go vet "$1" 2>&1

If non-zero exit: this counts as an attempt. Read the error, revert the test edit, refine, and retry (up to 3 total). After 3 failed compile attempts → outcome compile-fail → mark skipped.

f. Clean check (test passes on unmutated code, no collateral damage):

go test -v -json -count=1 "$1" > /tmp/clean.json 2>&1 || true
jq -r 'select(.Action=="fail" and .Test!=null) | .Test' /tmp/clean.json | sort -u > /tmp/clean.failed

• If the new test name appears in /tmp/clean.failed → outcome clean-fail → revert test edit, retry.
• If comm -13 /tmp/baseline.failed /tmp/clean.failed is non-empty (new failures other than ours) → outcome collateral-failure → revert test edit, retry.

g. Apply mutation.

Load references/mutation-transforms.md. Find the transform list for <type>. Read the source file. Compute the byte offset for (line, column) (1-based line; column is the 1-based byte offset within the line). Find the first transform whose from string appears at exactly that offset. If none match → ABORT this mutation, mark `skipped` with reason "transform mismatch" (do NOT write anything).

Before editing:

PRE_HASH=$(git hash-object "<source-file>")

Apply the edit with the Edit tool using a unique surrounding context (read enough bytes around the offset to make old_string unambiguous).

h. Mutant test run.

go test -v -json -count=1 "$1" > /tmp/mutant.json 2>&1 || true
jq -r 'select(.Action=="fail" and .Test!=null) | .Test' /tmp/mutant.json | sort -u > /tmp/mutant.failed
grep -q '^panic:' <(jq -r 'select(.Output!=null) | .Output' /tmp/mutant.json) && echo PANIC > /tmp/mutant.panic || true

Required outcome for success: new test name is in /tmp/mutant.failed, AND /tmp/mutant.panic is empty, AND comm -13 <(sort -u /tmp/baseline.failed <(echo "<new-test-name>")) /tmp/mutant.failed is empty (no unrelated new failures).

i. Revert mutation — ALWAYS attempt, even after a failed mutant run.

git checkout -- "<source-file>"
POST_HASH=$(git hash-object "<source-file>")
[ "$PRE_HASH" = "$POST_HASH" ] || { echo "FATAL: revert mismatch"; exit 1; }

If the hash check fails → STOP THE ENTIRE RUN, report the corrupt state to the user, do not touch state.toml further, do not commit anything.

j. Classify the result using references/outcomes.md:

• caught → mark tested, leave the new test in place, break out of the retry loop.
• mutant-not-caught / panic-in-output / collateral-failure → revert test edit, retry (up to 3).
• compile-fail / clean-fail → already handled above.
• After 3 failed attempts → mark skipped and write a skip annotation (step 8).

8. Skip annotation (only when marking skipped)

When a mutation can't be caught after retries, insert a comment line immediately before the enclosing function:

//gremlins:skip <TYPE> — exhausted retry budget, no generated test caught this mutation

Find the enclosing function by reading the source file and scanning backward from the mutation line for the nearest line matching ^func (top-level or method). Insert the comment line immediately above that line. Do not insert if a //gremlins:skip already exists on the line directly above the function.

9. State save

After every mutation (success or skip), rewrite .testweaver/state.toml atomically (temp file + rename). See references/state-schema.md for the format.

10. Finalize

If at least one mutation was marked tested in this run:

DATE=$(date +%Y-%m-%d)
PKG_SLUG=$(echo "$1" | tr '/' '-')
BRANCH="testweaver/${DATE}-${PKG_SLUG}"

# Collision check
SUFFIX=2
NAME="$BRANCH"
while git show-ref --verify --quiet "refs/heads/$NAME" \
   || git show-ref --verify --quiet "refs/remotes/origin/$NAME"; do
  NAME="${BRANCH}-${SUFFIX}"
  SUFFIX=$((SUFFIX + 1))
  [ "$SUFFIX" -gt 100 ] && { echo "too many collisions"; exit 1; }
done
git checkout -b "$NAME"

Verify git status --porcelain shows only test files and skip annotations. If anything else, abort and report — do not commit.

git add <each-test-file> <each-annotated-source-file> .testweaver/state.toml
git commit -m "testweaver: add mutation tests for $1"
git push -u origin "$NAME"
gh pr create \
  --base "$(git symbolic-ref refs/remotes/origin/HEAD | sed 's|refs/remotes/origin/||')" \
  --head "$NAME" \
  --title "testweaver: mutation tests for $1" \
  --body "$(printf 'Generated by /testweaver\n\n## Caught\n%s\n\n## Skipped\n%s\n' "$CAUGHT_LIST" "$SKIPPED_LIST")" \
  --label "testweaver"

Report the PR URL.

If zero mutations were caught: report what was tried and skipped, do not branch/commit/push.

Reporting back to the user

Final message must include:

• Mutations attempted, caught, skipped (counts + IDs).
• PR URL if created, otherwise reason no PR was opened.
• Any HARD-RULE aborts (corrupt revert, transform mismatch) — these are loud failures the user must see.