Bio Stats ML Reporting

Aggregate results, train ML models, and produce reports with validated references.

Instructions

1. Join outputs in DuckDB v1.1+ and build feature tables. Arrow / DuckLake integration is the recommended bridge into ML pipelines for large datasets.
2. Train baseline models and evaluate with cross-validation.

• CPU baseline: scikit-learn v1.5+ for linear/tree/clustering baselines; XGBoost v2.1.4+ for gradient boosting.
• GPU node available (CUDA): set device="cuda" on XGBoost (native since v2.0) by default. For sklearn-compatible estimators (random forest, k-means, PCA, UMAP), use RAPIDS cuML as a drop-in replacement and record the device in the run log.

1. Generate reports and validate references.
2. For exploratory omics projects, aggregate discovery evidence across the literature-derived analysis playbook, annotation, phylogenomics, viromics, and comparative-genomics outputs.
3. Comparative-axes rollup — join the per-axis comparison artifacts produced by upstream skills into a single comparative_axes_summary.tsv. The rollup must have one row per (query genome, axis) and include:

• genome-property frontier (size, gene count, etc. — link to relative_genome_metrics.tsv and genome_size_frontier.tsv)
• marker-gene census (link to marker_census.tsv)
• family copy-number expansions/contractions (link to family_copy_number_comparison.tsv and family_expansion_candidates.tsv)
• synteny / conserved neighborhoods (link to conserved_neighborhoods.tsv)
• non-coding RNA census (link to ncRNA_census.tsv)

Each row records observation, comparison baseline, literature reference, status (notable / conserved / artifact / negative), and a follow-up test.

1. Produce an interesting-findings section that ranks candidate discoveries relative to the literature-derived baseline and separates:

• strong candidates with multiple evidence types
• plausible candidates needing validation
• likely artifacts or conserved lineage features
• explicit negative findings where nothing notable was detected

1. Include the comparison baseline, literature context, confidence, and next discriminating analyses for each candidate.

Quick Reference

Input Requirements

Prerequisites:

• Tools available in the active environment (Pixi/conda/system). See docs/README.md for expected tools.
• Results tables and metadata are available.

Inputs:

• results/.parquet or results/.tsv
• metadata.tsv

Output

• results/bio-stats-ml-reporting/models/
• results/bio-stats-ml-reporting/metrics.tsv
• results/bio-stats-ml-reporting/comparative_axes_summary.tsv
• results/bio-stats-ml-reporting/discovery_summary.tsv
• results/bio-stats-ml-reporting/report.md
• results/bio-stats-ml-reporting/logs/

Quality Gates

• [ ] Model performance sanity checks pass.
• [ ] Reference validation passes.
• [ ] On failure: retry with alternative parameters; if still failing, record in report and exit non-zero.
• [ ] Verify input tables are readable and schema-consistent.
• [ ] Discovery summary joins candidate genes/features to annotation evidence, comparison baseline, literature context, and confidence.
• [ ] comparative_axes_summary.tsv covers all five mandatory axes (genome-property frontier, marker-gene census, family copy-number, synteny/neighborhoods, ncRNA census) for every query genome, with rows for axes that produced negative findings.
• [ ] Final report states what is interesting, what is conserved/expected, what is likely artifact, and what should be tested next.

Examples

Example 1: Expected input layout

results/*.parquet or results/*.tsv
metadata.tsv

Troubleshooting

Issue: Missing inputs or reference databases Solution: Verify paths and permissions before running the workflow.

Issue: Low-quality results or failed QC gates Solution: Review reports, adjust parameters, and re-run the affected step.