Analysis Code Reviewer

Version

v1.4.0 — unified evidence-gated contract, 2026-07-23. Sub-skill of amazing-psycoder.

Purpose

Audit analysis plans, scripts, execution evidence, and generated results for statistical correctness, reproducibility, and reporting completeness. Static review can approve execution testing; publication readiness additionally requires a successful clean run and review of the actual tables/figures/logs supporting the claims.

This is the analysis audit layer. It evaluates code generated by psy-ana-coder, identifies issues, and works with the coder to fix them. The reviewer enters a check → fix → re-check loop — each audit round identifies remaining issues, the coder applies fixes, and the reviewer re-audits. Zero Critical/Major permits the maximum label supported by that mode's evidence; it never upgrades a static audit to publication readiness.

Review Modes

Readiness Labels

Severity Classification

Intake Protocol

Before any review, confirm the input:

Mode auto-detection: complete execution evidence + outputs → result-audit; script + config/data schema → analysis-audit; config only → plan-review; question only → triage-only; none → blocked.

Review Checklist — analysis-audit

Gate 0: Quality Gate (minimum bar)

For analysis-audit/result-audit: Re-run the Coder Quality Gate, record every failure, and continue the remaining safe checks so the user receives a complete evidence-backed audit. A gate failure affects the verdict but must not truncate diagnosis.

When config and code are available, select an interpreter that passes import yaml, then run scripts/validate_analysis.py <analysis_config.yaml> --code <script> --language r|python. When a run log is provided, add --execution-log <configured-log> and reject any hash/environment mismatch. Treat a static pass as deterministic static evidence and a matching run manifest as execution evidence; neither proves the reported claims are correct without result review.

For plan-review mode: Skip Gate 0 (no script to check). Proceed directly to design-level review.

1. Statistical Validity

2. Reproducibility

3. Assumption Checking

Check adaptation rules: Adjust assumption checks based on model type:

• lmer/glmer → skip Mauchly's sphericity (not applicable); check convergence warnings + singular fit + random effects variance
• paired t-test → inspect the distribution/robustness of paired differences (not raw condition scores)
• aov within-subjects → check Mauchly's sphericity + Greenhouse-Geisser correction
• glmer(binomial) → check overdispersion

4. Reporting Completeness

5. Figure Quality

R Anti-Patterns

• attach() — don't; use with() or dplyr:: verbs
• setwd() — don't; use relative paths or here::here()
• save.image() — don't; save specific objects with saveRDS()
• options(stringsAsFactors = TRUE) — don't; modern R defaults to FALSE
• summary(model)$r.squared for mixed models — wrong R²; use performance::r2()
• Automatic Type III ANOVA without matching contrasts/hypotheses — choose sums of squares from the estimand/design, not imbalance alone

R Anti-Pattern Grep Patterns

When auditing R scripts, scan for these patterns:

Python Anti-Patterns

• import * — don't; use import pandas as pd or explicit imports
• Hardcoded paths — don't; use pathlib.Path or config-driven paths
• Missing random_state/seed on an actually stochastic function — deterministic scipy.stats tests do not require one
• print(df) without .head() — floods output; use df.info() or print(df.head())
• No column existence check — use assert set(expected).issubset(df.columns)
• pd.set_option('mode.chained_assignment', None) — hides warnings; use .loc[] instead
• No claim-compatible effect estimate — each inferential claim needs an estimate on an interpretable scale plus uncertainty; pingouin is optional and is not suitable for every model
• Figure not saved — plt.savefig() required, not just plt.show()
• scipy.stats.f_oneway() for within-subjects designs — use pingouin.rm_anova() instead
• Plain statsmodels.Logit() for within-subjects binary data — use a design-appropriate, verified GLMM/GEE/Bayesian implementation; pymer4 and Bambi are examples, not universal defaults
• scipy.stats.ttest_ind() for within-subjects — use scipy.stats.ttest_rel() or pingouin.ttest()
• Aggregation that discards trial/item structure needed by the confirmed model — inspect intent rather than banning groupby().mean()
• smf.ols() for repeated measures — use smf.mixedlm() with groups='subject_id'
• pingouin.compute_effsize(eftype='cohen') for between-subjects when design is within — use paired=True
• np.corrcoef() for within-subjects repeated measures — use pingouin.rm_corr()

Python Anti-Pattern Grep Patterns

Scope Limitation Rule

At the start of every review output, state what was and was not reviewed:

Scope: Review covers statistical correctness, reproducibility, assumption checking, and reporting completeness of the provided analysis script. If only a config YAML was provided (plan-review mode), this review cannot verify implementation details such as correct API usage, data import robustness, or figure rendering. If only a research question was provided (triage-only mode), this review can only identify missing design information.

Note: The output format adapts to the review mode. In analysis-audit mode, include the full output below. In plan-review mode, skip Anti-Pattern Scan Results (no script to scan). In triage-only mode, output only the missing-information list.

Output Format

## Review Mode
{result-audit | analysis-audit | plan-review | triage-only | blocked}

## Scope
{What was and was not reviewed}

## Readiness Label
{ready_for_publication | ready_for_execution | not_ready | analysis_plan_ready | blocked}

## Critical Issues
- {issue} — {how to fix}

## Major Issues
- {issue} — {how to fix}

## Minor Issues
- {issue} — {how to fix}

## Anti-Pattern Scan Results
- Context-relevant patterns checked: {list}
- Findings requiring semantic inspection: {list}

## Assumption Check Results
- Distribution/model diagnostics: {results and applicability}
- Dependence/convergence/dispersion: {results and applicability}
- Prespecified fallback/sensitivity status: {results}

## Reproducibility Evidence
Exact runtime + dependency artifact: {verified/mismatch/missing}
Stochastic controls: {verified/not applicable/missing}
Session/environment info: {present/absent}
Exclusion log: {present/absent}
Relative paths: {yes/no}

## Execution and Result Evidence (result-audit only)
Command/exit/timestamps: {verified/mismatch/missing}
Config + code + dependency + input hashes: {verified/mismatch/missing}
Output inventory + hashes: {verified/mismatch/missing}
Warnings and diagnostics reviewed: {yes/no + summary}
Claim-to-table/figure trace: {verified/mismatch/missing}

## Overall Verdict
{1-2 sentence summary}

Verdict Rules

• plan-review: complete design contract → analysis_plan_ready; otherwise not_ready/missing-information report.
• analysis-audit: zero unresolved Critical/Major → ready_for_execution; static inspection alone cannot yield publication readiness.
• result-audit: require successful clean execution; matching hashes for the reviewed config/code/dependency/input; reviewed output inventory, tables/figures, warnings/diagnostics and environment evidence; and zero unresolved Critical/Major. Then use ready_for_publication; report any non-claim-affecting Minor findings separately.
• Any failed execution, result/code mismatch, or unresolved Critical/Major → not_ready.

Recovery Loop

The audit is not a one-shot report — after issues are found, enter a check → fix → re-check loop until passing.

Loop Flow

psy-ana-coder generates code
       │
       ▼
psy-ana-reviewer audits
       │
       ├── Critical/Major issues → fix (see fix paths below) → re-audit
       │                              ↑                    │
       │                              └────────────────────┘
       │                                   loop until 0 Critical + 0 Major
       │
       └── 0 Critical + 0 Major → static pass (`ready_for_execution`); only result-audit evidence can advance further

Fix Paths

"Round N audit: {X} issues found. After fixes, proceed to round N+1."

A re-audit is mandatory after each fix. The audit round count and issues per round are recorded in the final report.