jedpsych-data-analysis — independently scanned and version-tracked by SaferSkills.
SaferSkills independently audited jedpsych-data-analysis (Agent Skill) and scored it 100/100 (green). The audit ran 55 deterministic rules across Security, Supply Chain, Maintenance, Transparency, and Community; it found 0 high-severity and 0 lower-severity findings. The full rule-by-rule trace and per-finding evidence are below. Free, methodology-open.
Findings & checks · 0 flagged
Every scanned point with the score it earned and what moved between them.
First recorded scan — no prior version to compare against.
The primary manifest — the file an agent reads to learn what this artifact does.
The Journal of Educational Psychology holds analyses to the standards of a rigorous psychological research journal operating in nested educational settings. The recurring requirements are: model the nesting (students in classes in schools), report effect sizes with confidence intervals that are educationally interpretable, test the mechanism (mediation/moderation), and disclose fully under JARS. Analysis scripts and data are expected to be shareable and reproducible.
account for students nested in classrooms/schools. Cluster-robust or random-effects inference is expected; ignoring clustering deflates standard errors and is a standard JEP rejection reason.
g, a multilevel d, R²/variance explained, or a growth-rate difference) with a confidence interval, and interpret it in learning terms (e.g., months of progress, percentile shift) — not just p-values and stars.
process, fit the mediation (with appropriate multilevel mediation methods) or moderation, not only the total effect.
exclusions/attrition (with reasons and counts), missing-data handling (e.g., FIML/multiple imputation), and model specification. Confirmatory vs. exploratory must be clearly separated.
correct for multiple comparisons across many outcomes; consider robustness to alternative specifications.
exclusions). Handle attrition and missingness with principled methods (FIML, MI) and report rates by arm.
A preregistered cluster-randomized reading-comprehension trial (48 classrooms, ~1,100 students). The confirmatory analysis is a two-level model with a pretest covariate and a preregistered mediation test.
Confirmatory (preregistered) — primary effect
Two-level model (students within classrooms), pretest-adjusted:
classroom-level treatment effect on transfer comprehension
g = 0.23, 95% CI [0.06, 0.40]; ICC = 0.14; ~2.0 months of progress.
Inference uses random classroom intercepts; SEs respect clustering.
Confirmatory (preregistered) — mechanism
Multilevel mediation: monitoring gain mediates ~40% of the effect,
indirect 95% CI [0.02, 0.13] (excludes 0).
Sensitivity: holds with/without the preregistered attrition exclusions
(g 0.23 → 0.21), and under FIML for missing posttests.
Exploratory (labeled): larger effect for initially low-comprehension
readers (ATI); reported as exploratory, flagged for future confirmation.Why this passes JEP scrutiny: the model respects nesting; the effect carries a CI and an educational interpretation; the mechanism is tested, not asserted; the sensitivity line pre-empts the "fragile-to- exclusions" reviewer; and the ATI is honestly demoted to exploratory.
| Reviewer pushback | What it signals here | JEP fix |
|---|---|---|
| "You ignored clustering" | deflated SEs from nesting | refit a multilevel/random-effects model; report the ICC |
| "Effect size, and what does it mean for learning?" | post-reform interpretability bar | add a CI and an educational metric (months/percentile) |
| "Mechanism untested" | total effect without theory | fit the preregistered multilevel mediation/moderation |
| "Which analyses were preregistered?" | forking-paths suspicion | give the disclosure table; relabel post hoc as exploratory |
| "How was attrition handled?" | missing-data validity | report rates by arm; use FIML/MI; show robustness |
pile of stars from a model that treated students as independent — the latter is a routine JEP reject.
progress, 95% CI [...]") to dichotomous "significant/not."
mediation/moderation test as a first-class result, not an afterthought.
Run the battery, don't just enumerate it. Full map: execution-with-mcp. JEdPsych mixes field/lab experiments and observational school data; multilevel (student-in-class-in-school) inference and many-outcome corrections matter most.
romano_wolf (step-down FWER) orbenjamini_hochberg — report the adjusted threshold.
oster_delta / sensemakr.wild_cluster_bootstrap (few clusters), twoway_cluster / conley;multilevel data → cluster at the right level.
audit_result(result_id) lists the missing checks and theexact suggest_function for each.
etable / did_summary_to_latex from the handle — no retyped numbers.Keep the decisive checks in the body and the exhaustive battery in the supplement. See the executed chain in the JF execution walkthrough.
【Model】multilevel / SEM / growth — nesting respected? [Y/N]
【Main result】effect size + CI + educational interpretation
【Mechanism】mediation/moderation tested as hypothesized? [Y/N/NA]
【Disclosure】N-determination + all exclusions/attrition + all measures (JARS)? [Y/N]
【Confirmatory vs exploratory】clearly separated? [Y/N]
【Reproducible】scripts + codebook + missing-data method? [Y/N]
【Next】jedpsych-tables-figures../../resources/external_tools.md — lme4/nlme, lavaan/Mplus, mediation, metafor, effectsize, missing-data tools../../resources/official-source-map.md — JARS statistical and disclosure requirements~30 seconds. Free. No account. Every finding cites a rule and a line of evidence.