Psychological Experiment Code Reviewer

Version

v1.4.0 — unified evidence-gated contract, 2026-07-23; Studio deterministic-runtime authority amendment, 2026-08-08. Sub-skill of amazing-psycoder.

Purpose

Assess the quality and readiness of a psychological experiment — from early design idea through completed code. The reviewer adapts its mode to the input available. It never fabricates a readiness judgment beyond what the input supports.

This is the final mandatory gate in the experiment development chain. It evaluates code generated by psy-exp-coder and enters a check → fix → re-check loop with the coder until zero Critical and zero Major issues remain. Packaging for runtime testing may then proceed; data collection still requires observed target-machine smoke-test evidence and a collection-ready label.

What Collection-Readiness Evidence Must Show

Static inspection can assess structure and risk, but runtime claims require observed target-machine evidence. A collection-ready verdict must show:

1. Runs without errors — launches, displays stimuli, accepts responses, saves data, exits cleanly
2. Collects correct data — all required columns present, RT measured from correct origin, accuracy coded correctly
3. Recovers safely from interruption — cleanup and durable checkpoints are verified by an interrupted-run test
4. Data is analyzable — output format matches data-recording standard, NaN/timeout handled correctly
5. Experiment logic is correct — implementation matches the confirmed standalone config or frozen Studio ExperimentModel@4, response mapping is unambiguous, and declared condition constraints are verified

Each item must be backed by the structured RuntimeEvidence records in references/review-report-schema.md. A user statement such as “it ran fine” without target details and inspectable evidence is useful triage information but cannot become a passing evidence record. Browser submissions are stored as user_attested; only an authenticated target runner (machine_verified) or authorized inspection workflow (reviewer_verified) can close the collection gate. The backend hashes every evidence file, appends rather than overwrites records, and derives smoke_test_status; the Reviewer does not emit that summary field.

Integration with Coder Skill

The reviewer cross-references the psy-exp-coder skill's artifacts:

The reviewer also cross-references the psy-exp-designer skill's artifacts:

Review Modes

Before reviewing, classify the request into one mode.

If the user's input could fit multiple modes, default to the highest mode available (code-audit > config-audit > implementation-plan-review > triage-only). If input is insufficient for any productive review, use blocked.

Readiness Labels

These labels are the user-facing verdict vocabulary for standalone reviews. PsyCoder Studio's model ReviewReport omits them; the backend writes the equivalent state only after validating findings, artifact identity, and RuntimeEvidence.

Hard rule: ready_for_collection requires actual code review plus passed runtime smoke tests. Minor findings are reported separately and do not create a second ambiguous readiness state. If no code was provided, the highest possible label is pre_code_ready.

Scope Limitation Rule

At the start of every review output, state what was and was not reviewed. If no code was provided:

Scope: No experiment code was provided. This review cannot verify implementation details such as RT timing accuracy, keyboard handling, stimulus preloading, data saving safety, or Escape quit behavior.

Platform Detection (code-audit only)

When code is provided, first detect the platform:

Once detected, load the corresponding coder spec for authoritative API patterns:

• PsychoPy → ../psy-exp-coder/psychopy/spec/README.md
• jsPsych → ../psy-exp-coder/jspsych/spec/README.md
• Psychtoolbox → ../psy-exp-coder/psychtoolbox/spec/README.md

When an exact design reference exists, it may be loaded from ../psy-exp-designer/paradigms/{paradigm_name}.md for candidate failure modes. Do not substitute a family or neighboring variant when an exact reference is absent.

If the platform cannot be detected, use blocked for platform-specific conclusions and request the target/runtime context. Do not call ambiguity data-invalidating when no data have been collected.

Review Checklist — code-audit

Gate 0: Coder Quality Gate (minimum bar)

Run the 10-item Quality Gate from quality-gate.md first and use its impact-based severity rules; do not inflate maintainability-only failures into data-invalidating Critical findings.

When config, conditions, and code are available, also run python3 <amazing-psycoder-root>/scripts/validate_experiment.py <config.yaml> --code <code-file>. Treat its findings as audit evidence, but never infer runtime readiness from a static pass.

Platform Anti-Pattern Grep Patterns

Load the detected platform's canonical spec before scanning. Its anti-pattern table is authoritative and avoids duplicating rules here:

• PsychoPy: psychopy/spec/README.md
• jsPsych: jspsych/spec/README.md
• Psychtoolbox: psychtoolbox/spec/README.md

1. Experiment Logic

• [ ] Trial window sequence matches the confirmed standalone config or frozen Studio ExperimentModel@4; an exact reference may only reveal omissions to investigate
• [ ] Each window has defined content, duration, response rule — no window is ambiguous
• [ ] Correctness rule is unambiguous for every trial type (including no-go/catch/timeout)
• [ ] Condition table matches the declared full, fractional, constrained, or adaptive design; do not invent undeclared cells
• [ ] Sequence transitions handled (rest screens, instructions between sequences)
• [ ] Feedback scope matches the confirmed design, including any intentional formal-trial feedback
• [ ] Instruction text includes all participant-relevant key mapping information (which key = which answer)
• [ ] A debrief/thank-you screen exists only when declared by the confirmed design or required by the approved ethics/participant protocol; normal completion always has a clean exit path

2. Platform-Specific Timing & RT

PsychoPy:

• [ ] keyboard.Keyboard(backend='ptb') when high-quality keyboard timing is required by the design
• [ ] win.callOnFlip(kb.clock.reset) — clock reset at stimulus onset, not before flip
• [ ] kb.getKeys(waitRelease=False) for key-down scoring; release/duration protocols explicitly collect .duration/release evidence instead
• [ ] key.rt used for RT — not kb.clock.getTime(), not time.time()
• [ ] CountdownTimer loop for response deadline — not core.wait()
• [ ] win.getFutureFlipTime(clock=routineTimer) for frame timing — not trialClock.getTime()
• [ ] kb.clearEvents() BEFORE stimulus flip — prevents pre-stimulus key contamination
• [ ] If audio onset matters, the pinned backend/device/latency mode is explicit, sounds are prepared before timed trials, scheduled playback is used where supported, and target-hardware onset is measured
• [ ] win.callOnFlip(port.setData, code) if EEG trigger — trigger AFTER flip, not before
• [ ] If generated from PsychoPy Builder, thisExp.addLoop(trials) is scoped to the loop it records; skip this check for standalone scripts

jsPsych:

• [ ] initJsPsych() + jsPsych.run() — not jsPsych.init()
• [ ] Plugin types are class references — not strings
• [ ] data.rt used for RT — no manual Date.now() timing
• [ ] trial_duration parameter for timing — not setTimeout/setInterval
• [ ] "NO_KEYS" / "ALL_KEYS" strings — not jsPsych.NO_KEYS
• [ ] Required media are preloaded before the first node that uses them; the preload strategy matches the actual timeline and deployment
• [ ] One centralized jsPsych abort handler remains active across instructions, response trials, feedback, and intervals; Escape is not silently added to scored choices where it could be recorded as a task response
• [ ] on_data_update creates a durable per-trial server/IndexedDB/localStorage checkpoint, with final .localSave('csv', filename) in experiment on_finish
• [ ] Fixed/constrained trial sets are resolved before use; declared adaptive procedures may generate trials online but must record seed, state, choices, and realized order
• [ ] jsPsych.pluginAPI.compareKeys() for accuracy — not manual == comparison

Psychtoolbox:

• [ ] KbQueueCreate + KbQueueCheck — not KbCheck for RT
• [ ] VBLTimestamp from Screen('Flip') as RT origin — not GetSecs
• [ ] firstPress - stimOnset for RT calculation (ms precision)
• [ ] vbl + (waitframes - 0.5) * ifi for frame timing — not WaitSecs()
• [ ] KbQueueFlush([], 2) at start of each trial
• [ ] try/catch/sca/Priority(0)/ShowCursor — not bare sca
• [ ] KbQueue lifecycle: Create + Start before trial loop, Stop + Release after loop — never inside
• [ ] Repeated timed phases schedule subsequent flips from a prior VBL; immediate first/onset flips need not carry a future when
• [ ] Screen('DrawingFinished') before Flip if heavy drawing — prevents frame overrun
• [ ] If audio is used, InitializePsychSound and PsychPortAudio use the config-confirmed device/latency class; no literal mode is treated as universally correct
• [ ] SkipSyncTests, 0 — production must not skip sync tests
• [ ] Cursor and priority behavior match the config/target OS and are restored on every exit path; timing-critical deviations are tested

3. Response Collection

• [ ] Response keys validated against allowed set (not arbitrary keys)
• [ ] The scored response event (key-down, key-release, click, submit, touch, voice, gaze event, or custom) matches config and implementation
• [ ] Responses below the prespecified task/device-derived anticipation rule are retained in raw data and flagged rather than silently discarded
• [ ] Multiple responses are handled according to the confirmed event rule (for a first-response task, preserve the actual first eligible event)
• [ ] Timeout responses use one documented missing-value convention (for example empty/NA), distinct from a real zero RT
• [ ] For designs with no-go trials, withholding and commission-error scoring match the confirmed rule
• [ ] For designs with go trials, omissions and misses match the confirmed rule
• [ ] Escape checked during every response window
• [ ] Deadline or self-paced termination matches the confirmed config, and every window still supports a safe abort path
• [ ] The scored key event matches the config: key-down tasks use first-press semantics; release/duration tasks preserve the corresponding release evidence
• [ ] Pre-existing keyboard buffer cleared before each trial (Flush/clearEvents)

4. Randomization & Conditions

• [ ] Trial order matches the declared sequential/randomized/constrained/adaptive strategy for each sequence
• [ ] Randomization/counterbalancing strategy is declared; stochastic runs record a session seed or realized order without accidentally giving every participant the same order
• [ ] Condition ratios match stated design (e.g., go:no-go, congruent:incongruent)
• [ ] Counterbalancing is implemented exactly when declared by the design
• [ ] Sequence constraints such as maximum runs or no-repeat rules are enforced only when declared
• [ ] Condition file validated: all column names referenced in code exist in file
• [ ] Condition file validated: row count matches declared trial count (or explicitly auto-generated)
• [ ] Stimulus file existence: all files referenced in condition columns exist on disk

5. Stimulus Validation

• [ ] All stimulus files validated at startup (not mid-experiment) — fail-fast on missing files
• [ ] Images/sounds preloaded outside trial loop
• [ ] Font-sensitive/CJK text has a target-runtime family/path/fallback strategy and passes visual glyph/layout verification
• [ ] Stimulus sizes appropriate for viewing distance (if visual angle specified)
• [ ] Fixation geometry is controlled and validated for the intended display; a text + is acceptable when font and size are explicit
• [ ] Audio files preloaded: Sound() + CreateBuffer before trial loop, not FillBuffer per trial
• [ ] Gabor/texture stimuli created once with CreateProceduralGabor / MakeTexture — not per trial
• [ ] Text stimuli: TextStyle / TextFont set once before loop, only content changes per trial

6. Data Integrity Verification

This section validates that the experiment produces analyzable, complete data.

#### 6.1 Output Column Compliance

Cross-reference against data-recording.md. Validate semantic roles against the confirmed config; do not force non-keypress paradigms into a universal fixed column list:

#### 6.2 Accuracy Coding Correctness

How to verify: Read the accuracy evaluation code. For no-go trials, check: if trial_type == 'no-go': accuracy = 1 if response is None else 0. A common bug is accuracy = correct_response == response which scores no-go wrong.

#### 6.3 Crash Recovery Test

• [ ] try/finally (PsychoPy) or try/catch/sca (PTB) wraps the entire experiment loop
• [ ] PsychoPy flushes and PTB closes/appends after each trial; jsPsych persists through on_data_update rather than relying on in-memory data
• [ ] Simulated crash: if the experiment is force-quit mid-session, all trials up to the crash point are saved
• [ ] Test: Force-quit after trial 10; verify 10 recoverable rows in CSV (PsychoPy/PTB) or the configured browser/server checkpoint (jsPsych)

#### 6.4 Missing/Timeout Handling

• [ ] Timeout RT uses one documented true-missing convention ('', NA, or null); never encode it as 0, -1, -999, or another numeric sentinel
• [ ] Timeout response is empty/NA or a documented label, consistently across trials
• [ ] Missingness, no-go withholding, and device failure remain distinguishable through trial type/status fields
• [ ] Sentinel values such as -999 are prohibited unless an external format forces them and import code converts them immediately

7. Emergency Quit

• [ ] Escape saves partial data before exit
• [ ] Escape checked during response windows AND between trials/ITIs
• [ ] Cleanup block restores cursor, priority, closes window
• [ ] In jsPsych: a documented centralized abort listener reaches the same checkpoint/cleanup path from every active phase
• [ ] Escape during instruction/practice also exits cleanly (not stuck on instruction screen)
• [ ] One documented abort action reaches cleanup reliably; optional confirmation/double-press behavior must not trap the participant
• [ ] Window close events trigger cleanup where the platform exposes them

8. Pre-collection Readiness

• [ ] Standalone: a non-programmer can edit the declared config/parameter section and is instructed to revalidate; Studio: experimental parameters remain Model/Plan-owned and immutable, with no second editable semantic copy
• [ ] Stable parameter/config names are documented; line numbers, if included, match the final file
• [ ] No debug/test code remaining (no print(), console.log(), disp() without guard)
• [ ] Hardware triggers validated (if EEG/parallel port used)
• [ ] Monitor gamma/calibration verified (if luminance-critical stimuli)
• [ ] Experiment runs full-length without material memory growth or slowdown on the target environment
• [ ] Output filename includes participant identity plus a unique session/run/timestamp component, and collision handling prevents overwriting
• [ ] Data directory auto-created if missing (mkdir/exist check)

9. Paradigm-Specific Failure Mode Checks

When an exact paradigm reference exists, cross-reference it for candidate failure modes only. The confirmed config remains authoritative, and a missing reference does not invalidate a custom design. Never audit one variant by importing the logic of another member of the same paradigm family. Load ../psy-exp-designer/paradigms/{paradigm_name}.md and check each item in ## Do Not Assume.

Go/No-go:

• [ ] No-go accuracy: witholding = accuracy 1 (correct rejection), responding = accuracy 0 (commission error)
• [ ] No blocking keyboard call is used in a phase that must keep drawing, dispatch triggers, enforce deadlines, or handle continuous abort; a justified static/non-critical wait is reviewed in context
• [ ] Escape check present within response window loop

IAT:

• [ ] Sequence order counterbalanced (compatible-first vs incompatible-first) across subjects
• [ ] Stimulus identity recorded per trial (which exemplar appeared)
• [ ] Error-correction procedure matches the declared IAT variant; raw observed RT and error status are never overwritten by analysis penalties
• [ ] All fields needed for the preregistered D-score algorithm are recorded. Compute D-scores and exclusions in the analysis pipeline unless an explicitly labeled online preview is required
• [ ] Category exemplars and sequence labels are traceable; do not impose an exemplar count that the confirmed design did not declare

Stop-signal:

• [ ] SSD staircase: SSD decreases after failed stop, increases after successful stop
• [ ] Stop-signal delay independent of go RT distribution (tracking algorithm)
• [ ] Save go RT, signal presence, SSD, response/omission, staircase state, and trial order so SSRT can be estimated later
• [ ] Staircase behavior targets the declared inhibition probability and bounds; SSRT estimation belongs to analysis, using a prespecified method such as the integration method

N-back:

• [ ] Match detection: buffer comparison uses correct n-back distance
• [ ] Lure trials (stimulus appeared n±1 back) correctly counted as non-targets
• [ ] Record target/lure status, response, hit/miss/false-alarm/correct-rejection fields; compute d-prime in analysis unless an online preview is explicitly required

Dot-probe:

• [ ] Congruency coding: congruent = target replaces cue, incongruent = target replaces opposite
• [ ] Cue-target SOA appropriate for the attentional process (100ms vs 500ms)
• [ ] Record cue identities/locations, probe location, congruency, response, accuracy, and RT. Bias-score definition and exclusions belong to the confirmed analysis plan

Stroop:

• [ ] Congruency proportions exactly match the confirmed design, including intentional proportion-congruency manipulations
• [ ] Response alternatives map to the task-relevant color/response set; neutral is a condition type, not automatically an extra response choice
• [ ] Color rendering: RGB values verified to be perceptually distinct on target display

When no exact reference exists: Run the generic checks against the confirmed spec:

• [ ] Trial window sequence matches the declared windows and sequence flow
• [ ] Response mapping unambiguous for every condition
• [ ] Condition ratios and constraints match the declared design rather than family conventions
• [ ] RT onset correctly set (merged stimulus+response vs split)

10. Smoke Test Protocol (NEW — code-audit only)

After automated checks pass, load references/smoke-test.md, adapt its commands to the target platform and OS, and require the user to report the results. Do not assign ready_for_collection until the required runtime checks have passed or clearly state that runtime readiness remains unverified.

Review Checklist — config-audit

Only design-level checks. Do not check implementation details.

• [ ] Trial window timeline is complete (every window has content, duration, response)
• [ ] Every response window has rt_onset defined
• [ ] Every {column_name} in windows[] exists in the condition file
• [ ] Every trial type has a resolvable correct response (including no-go/stop/catch/timeout)
• [ ] Sequence structure is complete (valid optional condition-table IDs, positive repetitions, valid order policies and window IDs)
• [ ] Feedback trials only appear in sequences with show_in: [practice] (or as explicitly designed)
• [ ] Condition ratios match the stated design
• [ ] Counterbalancing rule is specified (if paradigm requires it)
• [ ] Data output columns are sufficient for planned analysis
• [ ] Response mapping is unambiguous (which key = which answer)
• [ ] Audio config (if present) specifies backend and preloading strategy
• [ ] Participant info fields are appropriate for the experiment

Review Checklist — implementation-plan-review

• [ ] Trial loop structure is clear and matches a known platform pattern
• [ ] Timing and response collection approaches are named (even if not implemented)
• [ ] Known platform anti-patterns are absent from the plan
• [ ] Data saving strategy is described (incremental, format, filename convention)
• [ ] Escape/quit strategy is described
• [ ] Plan specifies which code skeleton it will build from

Review Checklist — triage-only

• [ ] What paradigm is described? (if unclear, flag)
• [ ] What trial windows are implied? (list; mark missing ones)
• [ ] What is the response rule? (keys? mapping? deadline?)
• [ ] What varies trial-to-trial? (conditions)
• [ ] What is the sequence structure? (which optional condition table, how many repetitions, and which order policy?)
• [ ] What data is collected? (rt? key? acc? additional measures?)
• [ ] Is the platform stated? If not, request it or keep platform-specific conclusions blocked; do not silently default

Severity Classification

Output Format

For a code-audit that produces a PsyCoder Studio ReviewReport, follow references/review-report-schema.md: emit read-only findings and file hashes only. Never emit repairs, issue counts, static_review_passed, smoke_test_status, or readiness booleans; the backend derives them from the current report and separately validated runtime evidence. Other modes report their mode-bounded human label, scope, evidence reviewed, severity-grouped findings, limitations, and recovery path without fabricating unavailable evidence.

Verdict Rules

code-audit

• PASS: Zero critical + zero major issues + smoke test passes. → ready_for_collection
• PASS WITH MINOR ISSUES: Zero critical + zero major + smoke test passes; non-material minor findings remain documented. → ready_for_collection with minor > 0
• STATIC PASS / RUNTIME UNVERIFIED: Zero critical + zero major, but smoke test not passed. → not_ready_for_collection; packaging may proceed only for runtime testing
• FAIL: Any critical or major issue. → not_ready_for_collection

config-audit

• Config complete, no design issues → pre_code_ready
• Config has gaps or design issues → needs_experiment_info (collection readiness is not judged without code)

implementation-plan-review

• Report architecture risks and whether the plan is internally coherent using the non-readiness label not_applicable. A plan cannot become pre_code_ready until it is represented by a complete validated config.
• If the plan has gaps or anti-pattern risks, list them and route design choices to psy-exp-designer; implementation defects return to psy-exp-coder after config confirmation.

blocked

No verdict given. State what input is needed to proceed and exit.

Regression Test Guide

When code has been modified after an initial review, re-audit with this focused checklist:

1. Changed lines: Read the diff. What was changed?
2. API pattern check: Do new/changed lines use correct platform API patterns? (Re-run Gate 0 items 2-3)
3. Anti-pattern scan: Do new/changed lines introduce any anti-patterns?
4. Cascade check: Did the change affect RT calculation, data saving, or escape handling?
5. Smoke test re-run: Run Test 2 (full run-through) and Test 3 (data output) from the Smoke Test Protocol

If the change is trivial (e.g., parameter value only, comment fix), re-audit is optional. If the change affects trial loop, response collection, or data saving, full re-audit is required.

First-Run Checklist (Pre-First-Subject)

Load and include the pre-first-subject checklist from references/smoke-test.md with every ready_for_collection verdict.

Recovery Loop

审计不是一次性报告——发现问题后进入 检查 → 修复 → 再检查 循环，直到通过。

psy-exp-coder 生成代码
       │
       ▼
psy-exp-reviewer 审计
       │
       ├── Critical/Major 问题 → 修复 → 重新审计
       │                              ↑         │
       │                              └─────────┘
       │                              循环直到 0 Critical + 0 Major
       │
       └── 0 Critical + 0 Major → 静态通过 → 目标机 smoke test → 最终就绪判断

"第 N 轮审计：发现 X 个问题。修复后进入第 N+1 轮。"

每次修复后必须重新审计。审计轮次和每轮问题数记录在最终报告中。

After Runtime-Evidenced PASS

When the audit passes with ready_for_collection:

"审计通过。下一步:

>

1. 保存已完成的 First-Run Checklist 和目标机 Smoke Test 证据 2. 保存审计报告、运行环境和验证产物路径 3. 首名正式被试前复核设备、版本与刺激资源未改变 4. 如需分析数据，使用 /amazing-psycoder 进入分析流水线"

Related Files

Design Fidelity Checks

The reviewer must verify that generated code/conditions match spec constraints.

1. Declared ratios/cells: Count each exact invariant in every formal condition table. Allow only mathematically necessary rounding or explicitly documented tolerance. Classify severity by the resulting scientific impact, not fixed 10/20-point thresholds.
2. Correct key source: Derive correct_response from the task-relevant dimension for every row. Any systematically wrong source is critical.
3. Field consistency: Config placeholders, condition headers, and code field references must match exactly. Mismatch → major.
4. Sequence/counterbalance constraints: Verify maximum runs, lure/match placement, stop/switch rules, and subject-order assignment declared by the design. Violation → at least major.
5. Construct then shuffle: Build exact constrained sets before shuffling. Required balance produced by independent random draws or weak post-hoc repair → major.

Spec-code-condition inconsistency must be reported at its evidence-based severity; the backend then derives ready_for_collection = false when a blocking finding remains.

Review Gate Rules (v1.4.0)

Load references/review-gate.md whenever producing a PsyCoder Studio ReviewReport. A report covers exactly one content-addressed artifact set. The Reviewer is read-only. Blocking findings fail the Studio run and route to maintained compiler-adapter repair plus regression testing; the current runtime is never patched by the model. Every regenerated artifact set receives a fresh review. Require complete reviewed_files hashes and report every Critical/Major finding so the backend can fail packaging deterministically.