MCP Evaluation Harness

Runs an A/B comparison of the code-index MCP server against the default Read/Grep/Glob baseline on the same task, then reports tokens, tool calls, latency, and a judge-scored quality verdict.

Preconditions

Before doing anything else, verify all three:

1. The code-index MCP is connected — mcp__code-index__* tools must be visible in the current toolset. Check with /mcp if unsure.
2. .claude/index.db exists and was modified within the last 24 hours. Run ls -la .claude/index.db and check mtime.
3. The task is read-only. Phrasings like "find", "where", "explain", "trace", "what calls", "how does" are fine. Anything that says "edit", "fix", "add", "implement", "rename", "refactor" is NOT — running the same edit twice in parallel corrupts the working tree.

If any precondition fails, stop and tell the user which one. Do not proceed with workarounds.

Procedure

Step 1 — Lock the task prompt

Capture the user's task verbatim into $TASK. Do not rephrase, expand, or "improve" it. Both agents must receive byte-identical wording, or the comparison is invalid.

Step 2 — Spawn both research agents in parallel

Issue both Task tool calls in a single assistant message so they run concurrently. Sequential runs double wall time and skew the latency numbers.

• Agent A: subagent_type: "eval-with-mcp", prompt: $TASK
• Agent B: subagent_type: "eval-without-mcp", prompt: $TASK

These sub-agents are defined in .claude/agents/eval-with-mcp.md and .claude/agents/eval-without-mcp.md. Their tool restrictions are enforced at the framework level — do not pass extra tools.

Step 3 — Extract metrics from each result

Each sub-agent terminates its response with a fenced JSON block:

{
  "tool_calls": <int>,
  "files_inspected": ["path1", "path2"],
  "answer": "<final answer>",
  "confidence": 0.0
}

Parse this JSON. Then from the Task tool's metadata, capture:

• total_input_tokens (input + cache_read summed)
• total_output_tokens
• wall_time_seconds

If a sub-agent did not emit the JSON block, record parse_failure: true for that side and continue. Do not retry — the failure is signal.

Step 4 — Run the judge

Spawn the eval-judge sub-agent (no tools, text-only reasoning). Pass it:

• The original $TASK
• Both answer strings, clearly labeled agent_a and agent_b
• Both files_inspected lists

The judge returns a JSON verdict with per-dimension scores and a winner.

Step 5 — Render the report

Produce this table for the user. Use Sonnet pricing ($3/M input, $15/M output) unless the user has indicated Opus or Haiku.

## MCP Evaluation Report

**Task**: <verbatim>
**Repo**: <git rev-parse --short HEAD>
**Index age**: <hours since .claude/index.db mtime>

| Metric                | With MCP  | Without MCP | Delta            |
|-----------------------|-----------|-------------|------------------|
| Input tokens          |           |             |                  |
| Output tokens         |           |             |                  |
| Total tokens          |           |             | -X (-Y%)         |
| Tool calls            |           |             |                  |
| Files inspected       |           |             |                  |
| Wall time             |           |             |                  |
| Correctness (1–5)     |           |             |                  |
| Specificity (1–5)     |           |             |                  |
| Completeness (1–5)    |           |             |                  |
| Hallucination safety  |           |             |                  |

**Judge verdict**: <a_wins | b_wins | tie>
**Reasoning**: <judge's reasoning, 1–2 sentences>

**Cost (this task)**: $<a> vs $<b> — savings $<delta>
**Projected monthly** (30 similar tasks/day): $<a × 900> vs $<b × 900>

### Answer excerpts
**With MCP** (first 300 chars): <…>
**Without MCP** (first 300 chars): <…>

Anti-patterns

• Never run on edit tasks. Two parallel agents editing in parallel corrupts the tree. Refuse, or convert to read-only ("describe the changes you would make instead of making them").
• Never rephrase the task between agents. Identical strings, or the comparison is invalid.
• Never give the judge tools. A tool-enabled judge re-investigates and biases the verdict toward whichever agent's evidence it can verify.
• Never run the agents sequentially. Parallel Task calls in one message — anything else skews wall time.
• Never declare a single run conclusive. State variance in the report. For real conclusions: 3+ varied tasks, aggregate.
• Never relax the without-MCP agent's constraints to "help" it. If it refuses or fails because it lacks the index, that IS the result.

Calibration

The MCP wins on exploration — "where is X", "what calls Y", "trace the flow of Z". It ties or loses on lookup — tasks where the file path is already known, or where one Grep nails it. A balanced eval set must include both. If every task is exploratory, the MCP looks better than its true average value.

Suggested representative task mix for a fair eval (run 5):

1. Pure discovery: "where do we handle authentication"
2. Symbol trace: "what calls parseJWT"
3. Refactor planning: "what's the blast radius of changing the User model"
4. Bug hunt: "find the code that handles 429 responses"
5. Known-path lookup: "explain what auth/jwt.ts does" — should be near-tie, sanity check

Failure modes

• Sub-agent times out → record as failure for that side. Do not retry.
• Sub-agent refuses ("I cannot do this without Read") → that IS the data point. Judge scores it accordingly.
• `code-index` MCP disconnected mid-run → abort and report. Don't fall back silently.
• Index stale (chunks reference deleted files) → abort and tell the user to reindex.
• Judge produces non-JSON → re-invoke the judge once with "respond ONLY with the JSON block, no preamble". If still fails, present the raw judge output to the user with a note.