cogpsych-data-analysis — independently scanned and version-tracked by SaferSkills.
SaferSkills independently audited cogpsych-data-analysis (Agent Skill) and scored it 100/100 (green). The audit ran 55 deterministic rules across Security, Supply Chain, Maintenance, Transparency, and Community; it found 0 high-severity and 0 lower-severity findings. The full rule-by-rule trace and per-finding evidence are below. Free, methodology-open.
Findings & checks · 0 flagged
Every scanned point with the score it earned and what moved between them.
First recorded scan — no prior version to compare against.
The primary manifest — the file an agent reads to learn what this artifact does.
Cognitive Psychology holds analyses to a model-based standard: fit the formal model, compare it to rivals with principled criteria, demonstrate that parameters and models are recoverable, use mixed models or hierarchical Bayesian estimation where the design demands it, and report effect sizes with uncertainty for behavioral results — all regenerable from deposited code. This is the experiment-to-model-fit loop that defines the venue.
matched flexibility; compare with AIC/BIC, cross-validation, or Bayes factors as appropriate, and say what the comparison licenses.
parameters from simulated data) and model recovery (does the comparison criterion pick the generating model) — without these, a fit edge is not interpretable.
(generalized) linear mixed models; for cognitive models, hierarchical Bayesian estimation pools strength across participants. Justify the structure; don't aggregate away the variance.
confidence/credible intervals for key behavioral results, not just p-values and stars.
exploratory model exploration; do not present a post hoc winning model as predicted.
fit, figure, and table in a fresh session (see cogpsych-open-science-and-transparency).
and exclusion choices; report sensitivity, not a single fragile fit. Report convergence diagnostics (e.g., R-hat, ESS) for Bayesian models.
A preregistered three-experiment recognition-memory program fitting UVSD vs. DPSD to confidence-ROC data.
Model comparison (preregistered) — pooled across Exps 1-3
Fit (hierarchical Bayesian, matched flexibility):
UVSD favored: dBIC = 14 vs. DPSD; Bayes factor ~ 30 in favor of UVSD
Recovery (required): parameter recovery good (recovered d', sigma within
credible intervals); model recovery ~ 92% correct at the design's N/trials
Diagnostic signature: z-ROC slope 0.78, 95% CrI [0.72, 0.84], and linear
(no reliable curvature) — the qualitative pattern UVSD predicts and DPSD
forbids, consistent across all three experiments
Behavioral effect (mixed model)
List-strength manipulation on d': b = 0.31, 95% CI [0.18, 0.44]
Exploratory (labeled)
A small response-bias drift surfaced post hoc; reported as exploratoryWhy this passes Cognitive Psychology scrutiny: the model is compared (not just fit), recovery makes the comparison interpretable, the qualitative signature corroborates the fit index, hierarchy respects subject/item variance, and the exploratory drift is honestly demoted.
| Reviewer pushback | What it signals here | Cognitive Psychology fix |
|---|---|---|
| "You only fit your model" | one-model storytelling | fit the rival under matched flexibility; report AIC/BIC/BF and what it licenses |
| "Better fit may be overfitting" | flexibility imbalance | add model recovery + cross-validation; penalize complexity |
| "Can you recover these parameters?" | identifiability doubt | run and report parameter + model recovery simulations |
| "Aggregated means hide variance" | wrong error structure | refit with crossed-random-effects mixed model / hierarchical Bayesian |
| "Is this the model you predicted?" | post hoc selection | pre-commit the comparison; relabel post hoc fits exploratory |
| "I can't rerun your fits" | reproducibility gate | ship seeded model code + a fresh-session run log |
good index but no rival and no recovery is not persuasive.
the signature the rival forbids.
bias parameter estimates; use mixed/hierarchical models and justify the random-effects structure.
reproducible evidence.
Run the battery, don't just enumerate it. Full map: execution-with-mcp. Cognitive Psychology is experimental — within-subject designs and mixed models dominate; report the model, the effect size, and multiple-comparison control.
romano_wolf (step-down FWER) orbenjamini_hochberg — report the adjusted threshold.
oster_delta / sensemakr.wild_cluster_bootstrap (few clusters), twoway_cluster / conley;multilevel data → cluster at the right level.
audit_result(result_id) lists the missing checks and theexact suggest_function for each.
etable / did_summary_to_latex from the handle — no retyped numbers.Keep the decisive checks in the body and the exhaustive battery in the supplement. See the executed chain in the JF execution walkthrough.
【Model comparison】rivals fit under matched flexibility + criterion (AIC/BIC/BF)? [Y/N]
【Recovery】parameter + model recovery reported? [Y/N]
【Hierarchy】mixed model / hierarchical Bayesian where apt + diagnostics? [Y/N]
【Behavioral effects】effect sizes + intervals? [Y/N]
【Confirmatory vs exploratory】separated? [Y/N]
【Reproducible】seeded code + data dictionary + fresh-session check? [Y/N]
【Next】cogpsych-tables-figures../../resources/external_tools.md — modeling, model-comparison, lme4/brms/Stan, JAGS, recovery simulation../../resources/official-source-map.md — statistical and modeling expectations~30 seconds. Free. No account. Every finding cites a rule and a line of evidence.