Ceres Publish Dataset — Monthly HuggingFace Export

Recurring monthly workflow to export the Ceres open data index to HuggingFace as a curated Parquet dataset.

Tracking issue: https://github.com/AndreaBozzo/Ceres/issues/89

Checklist

1. Run data quality checks (noise, duplicates, new portals)
2. Export: ceres export --format parquet --output ~/ceres-open-data-index
3. Remove any stale per-portal parquet files from previous exports (e.g. renamed portals)
4. Update README.md dataset card (counts, portal table, snapshot date, bias notes)
5. Version the snapshot (commit + tag)
6. Push to HuggingFace
7. (Optional) Regenerate HuggingFace Space visualization

Export Command

ceres export --format parquet --output ~/ceres-open-data-index

The export streams all non-stale datasets from PostgreSQL, applies curation, and writes:

• all.parquet — Complete flattened dataset
• data/<portal-name>.parquet — Per-portal subsets
• metadata.json — Snapshot metadata with counts and portal breakdown

Portal names are resolved from ~/.config/ceres/portals.toml. Portals not in the config fall back to hostname-based naming (e.g. https://data.gov.ro becomes data-gov-ro).

The export takes ~30-40 minutes for 900k+ datasets due to JSONB flattening.

Export Schema (14 columns)

Curation Rules (applied automatically during export)

• Noise filter: Removes datasets where title < 5 chars, description is empty, or title contains "test"/"prova"/"esempio" (case-insensitive substring match)
• Duplicate flag: Same title (case-insensitive) across different portals sets is_duplicate=true. Duplicates are kept, not removed. SQL: SELECT LOWER(title) FROM datasets GROUP BY LOWER(title) HAVING COUNT(DISTINCT TRIM(TRAILING '/' FROM source_portal)) > 1
• Metadata flattening: Tags from metadata.tags[].name, organization from metadata.organization.title, license from metadata.license_title

HuggingFace Dataset Repository

• Location: ~/ceres-open-data-index
• Remote: https://huggingface.co/datasets/AndreaBozzo/ceres-open-data-index
• Branch: main
• LFS: *.parquet tracked via Git-LFS (configured in .gitattributes)

Commit and push workflow

cd ~/ceres-open-data-index
git add all.parquet data/ metadata.json README.md
git commit -m "vN: Month Year export (X portals, Yk datasets)"
git tag vN
git push origin main
git push origin vN

README Dataset Card — What to Update

The README at ~/ceres-open-data-index/README.md uses HuggingFace dataset card format with YAML frontmatter. Update these sections:

1. YAML frontmatter: language list (add new language codes), size_categories, tags
2. Opening line: Total datasets count, portal count, country count
3. Files section: Row count, file count
4. Portal table (### Splits by portal): Add new portals, update all counts, sort by count descending
5. Curation section: Noise filtering count, duplicate flagging count
6. Update frequency: Snapshot date
7. Known biases: Geographic skew percentages, language distribution, portal selection notes
8. Citation block: Snapshot date, portal count

All numbers come from metadata.json (generated by the export).

Export History

Key Files

HuggingFace Space (Optional)

The visualization dashboard at https://huggingface.co/spaces/AndreaBozzo/Ceres is a static Plotly site generated from the exported data.

• Location: ~/Documenti/Ceres(huggingfacespace)
• Generator: ~/Documenti/open-data-galaxy/galaxy_constellation.py
• Regenerate after pushing the dataset, then push the Space repo separately

Pre-export Data Quality Checks

# Overall stats
ceres stats

# Check noise candidates in DB (optional)
docker exec ceres_db psql -U ceres_user -d ceres_db -c "
  SELECT COUNT(*) FILTER (WHERE LENGTH(title) < 5) as tiny_titles,
         COUNT(*) FILTER (WHERE description IS NULL OR TRIM(description) = '') as empty_desc,
         COUNT(*) FILTER (WHERE LOWER(title) LIKE '%test%' OR LOWER(title) LIKE '%prova%' OR LOWER(title) LIKE '%esempio%') as noise_titles
  FROM datasets WHERE NOT is_stale;
"