OctoPerf — Validation-failure triage

When a Virtual User validation finishes with many failing actions, reading each failure detail one by one wastes context window and time. This skill groups failures by root cause, drills into one representative per group, and maps each group to the correct fix.

When this applies

• A validate_virtual_user run finished with finished=true and at least one failing action.
• The user wants to know what's wrong, not just retry blindly.

Steps

1. Get the failure index — no bodies yet

mcp__octoperf__get_virtual_user_validation_index(virtualUserId)

The index returns one entry per validated HTTP action — actionId, success/total counts, successTimestamps and failedTimestamps. For triage, focus on the entries whose failedTimestamps is non-empty (the actions that failed at least one run); entries with only successTimestamps passed every run. Do not fetch failure details yet — the index alone is usually enough to classify.

(A successTimestamps entry is also a handle to read a passing run's body: pass it to fetch_validation_http_body with kind=VALIDATION_RESPONSE — e.g. to inspect what a Debug-style action actually captured on a green run.)

OctoPerf's "KO" rule isn't just "non-2XX". A sample is KO when:

This matters when classifying: a "successful" 200 against a recorded 302 is a real bug, not a false positive — the VU is hitting a different code path than when it was recorded.

Caveat — ResponseAssertion overrides the matrix. A ResponseAssertion attached to the action can mark a sample KO even when the matrix above says OK (e.g. status 200 against recorded 200, but the body contains the assertion's pattern — typical for forms that re-render with an error message on a 200). Check for assertions on the failing action before assuming an HTTP-level mismatch. The assertion's type field also controls scope:

• REQUEST_ONLY — only the parent sample is checked.
• REQUEST_AND_SUBREQUESTS — parent and embedded resources. Matters when downloadResources=true: a 404 on an embedded CSS can fail the parent.
• SUBREQUESTS_ONLY — only embedded resources.

1b. Partial failures (success < total)

When the index entry shows success < total (e.g. {"success": 2, "total": 3, "failedTimestamps": [...]}), the action failed on some iterations but passed on others. Common patterns:

• Bad row in a CSVVariable — one iteration picks a row that

doesn't pass server-side validation (e.g. a wrong-password row, a malformed account id, a stale token). Cheapest possible fix: edit the CSV. Investigate this before correlation when the VU uses a CSVVariable.

• Race condition — uncommon in 1-user validation but possible if

the VU has internal LoopContainerAction or shared variables.

• CSV exhausted with EOF = StopVU — iterations past the CSV length

never run; not strictly a failure but success < total reflects it.

failedTimestamps lists every failed iteration's epoch-ms. Pass the exact timestamp to get_validation_failure_detail to pin the failing iteration instead of getting a random one — critical when debugging CSV-driven flakiness so you see the bad row, not a good one.

2. Group by category

Bucket the failing actions into a small number of categories. The common ones:

A useful heuristic: if 80% of failures fall in one category, fix that first and re-validate before investigating the rest. Most of the long tail clears once the dominant root cause is resolved.

3. Confirm each category against one representative

For each group, pick the first failing action and fetch its detail:

mcp__octoperf__get_validation_failure_detail(virtualUserId, actionId)

The detail returns the four HTTP entities (sent/received request and response). For very large bodies, you can pull a single one with mcp__octoperf__fetch_validation_http_body(...). Its kind parameter (RECORDED_REQUEST / RECORDED_RESPONSE / VALIDATION_REQUEST / VALIDATION_RESPONSE) lets you isolate the recorded-vs-replay diff side-by-side — useful when the request/response is several MB and a single side fits in your context but not both.

Read each detail with a specific question in mind based on the category:

• Auth/state: is there a token in the previous response that should have been re-sent here? → Auto-correlation skill.
• Variable/data: does the request body contain a literal value that should be a ${variable} reference? → list_variables → create_*_variable → re-import or edit.
• HTTP server config: is the request reaching the server at all? Timing out? Hitting the wrong host? → list_http_servers_by_project → update_http_server.
• Body mismatch: does the recorded request body match what the API actually expects today? If not, the recording is stale.
• Assertion failure: is the matched pattern (e.g. "Signon failed" substring) a symptom of an upstream wrongness (auth failed, validation error re-rendered in the page) or an overly-strict assertion? Fix the upstream cause first; loosen the assertion's pattern only if the response is genuinely correct.
• Missing dependency: is the requested resource (/api/orders/12345) one that the recording created earlier, but whose id is now stale? → correlation, or use a CSV variable.

3b. Inspect live variable state with a DebugAction

When the category is Auth/state or Variable/data, the decisive question is usually "what value did the variable actually hold at this point in the flow?" — did an extractor capture nothing (empty), capture the wrong substring, or capture the right value that the server then rejected? The HTTP entities only show what was sent; they don't show the variable table.

Insert a DebugAction (JMeter's Debug Sampler) right after the action that populates the variable, then re-validate and read its body:

mcp__octoperf__patch_virtual_user(virtualUserId, ops=[{
  "op": "add",
  "path": "/children/<index>",
  "value": {
    "@type": "DebugAction",
    "id": "<fresh-uuid>",
    "enabled": true,
    "variablesDisplayed": true
  }
}])

Two payload gotchas: the body dumps the variable table only when variablesDisplayed: true — omit it and the response comes back empty. And DebugAction requires a non-null id (supply a fresh UUID) and has no name field — don't add one. propertiesDisplayed / systemPropertiesDisplayed stay false unless you also want the JMeter property dump.

A DebugAction always succeeds, so it never shows up in failedTimestamps — find its run under successTimestamps in the index, then pull its body:

mcp__octoperf__get_virtual_user_validation_index(virtualUserId)      # grab the DebugAction's successTimestamp
mcp__octoperf__fetch_validation_http_body(
  projectId, virtualUserId, debugActionId, timestamp, kind=VALIDATION_RESPONSE)

The body is a dump of every JMeter variable in scope at that point:

JMeterVariables:
JMeterThread.last_sample_ok=true
slideTitle=Sample Slide Show
slideTitle_matchNr=1
...

How to read it:

• `var=<value>` + `var_matchNr=1` → the extractor matched once and

captured the value. The correlation works; if the request still fails, the value is being rejected (stale, wrong scope), not missing.

• `var=` empty, or the variable absent entirely → the extractor

matched nothing. The regex/JSONPath is wrong or runs against the wrong response. → fix the extractor (patch_virtual_user) or the correlation rule.

• `var_matchNr` > 1 → multiple matches; the wrong occurrence may be

injected downstream. Pin the match number.

Remove the DebugAction once diagnosed (patch_virtual_user with an op: remove) — it's a diagnostic probe, not part of the script.

4. Apply ONE fix, then re-validate

Resist the urge to fix three things at once. Apply the fix that should clear the largest category, then:

mcp__octoperf__validate_virtual_user(projectId, virtualUserId, providerId, location, iterations=1)
mcp__octoperf__get_virtual_user_validation(projectId, virtualUserId)  # poll — see octoperf-async-polling

Re-fetch the failure index. Verify:

• Cleared: the target category is gone. Move on to the next.
• Reduced: some failures of the same category cleared, others didn't. The fix was partial — refine it (e.g. add a more specific correlation rule).
• Unchanged or worse: the fix was wrong; revert it (update_* or delete_*) before trying another.

5. Engine-level failure (no index, no HTTP entities)

If get_virtual_user_validation_index comes back empty but the run is still marked failed/aborted, the validation engine itself crashed (JMeter OOM, missing Playwright dependency, bad locale, …) — there are no HTTP samples to read. (An empty index now means no validated samples at all: a VU that ran and passed lists every action with only successTimestamps, so a truly empty result points at the engine, not a clean pass.) The validation run produces a benchResultId (returned by validate_virtual_user) which backs the same log storage as a real bench run, so:

mcp__octoperf__list_bench_result_files(benchResultId)
mcp__octoperf__read_bench_result_file_lines(benchResultId, "jmeter.log")

surfaces the engine logs (and any Playwright trace / screenshot / HAR the engine left behind). Read the tail of jmeter.log first — startup errors are usually within the first 50 lines and fatal errors within the last 50.

Log retention. Validation log files are erased 7 days after the run, or as soon as the user leaves the design screen. Old validation runs may no longer have logs — call out the freshness window before promising a re-read.

For binary artefacts (Playwright trace.zip, screenshots .png, HAR archives) the line-based reader returns garbage. Use the binary-aware tool instead:

mcp__octoperf__download_bench_result_file(benchResultId, filename)
# returns { url, method: "GET", expiresAt, instructions }

GET url directly with your code interpreter (single-use token, valid ~5 minutes) and inspect the bytes locally (e.g. unzip -p trace.zip trace.trace). This is especially valuable for Playwright VU failures — the JMeter wrapper log only sees the spawn, the actual selector miss / timeout / navigation abort lives in the trace.

6. Stop conditions

Stop and surface to the user when:

• Zero failures → VU is ready. Offer to run_scenario.
• All remaining failures are 5xx → it's the target environment, not the VU. Hand back with the list.
• After two rounds of fixes the failure count plateaus → there's a structural issue (wrong recording, target API changed, …). Hand back with the diagnosis.

Sanity-check output reference

sanity_check_virtual_user runs before the first validation. Its output is a flat list of (level, message). ERROR entries block validation, WARNING / INFO entries don't. Mapping the canonical messages to fixes:

Apply the ERROR fixes first — validation is blocked until those clear.

Anti-patterns

• Don't fetch failure details for every action. The index is enough to classify; details are for confirmation, not bulk inspection.
• Don't `run_scenario` to debug. Validation is the right tool — it's cheap, captures full HTTP. A load test gives you metrics, not bodies.
• Don't sanity-check after validating. sanity_check_virtual_user is a static check; run it before the first validation run, not after. If it would have caught the issue, you wasted a validation cycle.
• Don't edit the VU silently. Summarize what fix you're about to apply and confirm with the user before any delete_* or destructive change. For irreversible tree edits (patch_virtual_user, applying correlations), snapshot first with backup_virtual_user(virtualUserId, label="pre-triage") — there's no VU versioning to undo a bad patch.
• Don't omit `enabled` when adding an action. When a patch_virtual_user op adds a node to children, every boolean defaults to false if absent — so an action added without "enabled": true is created disabled and silently does nothing at run time, with no validation error. Always set "enabled": true explicitly on inserted actions. The JSON field is enabled, not isEnabled.

See also

• octoperf-auto-correlation — for the "auth / state" category.
• octoperf-scenario-diagnosis — for diagnosing problems that appear under load but not in validation.
• octoperf-async-polling — sleep cadence and terminal conditions for get_virtual_user_validation / get_bench_result.