curate-h5ad — independently scanned and version-tracked by SaferSkills.
SaferSkills independently audited curate-h5ad (Agent Skill) and scored it 100/100 (green). The audit ran 55 deterministic rules across Security, Supply Chain, Maintenance, Transparency, and Community; it found 0 high-severity and 0 lower-severity findings. The full rule-by-rule trace and per-finding evidence are below. Free, methodology-open.
Findings & checks · 0 flagged
Every scanned point with the score it earned and what moved between them.
First recorded scan — no prior version to compare against.
The primary manifest — the file an agent reads to learn what this artifact does.
Curate the h5ad file at absolute path: $ARGUMENTS
Pass an absolute path to the .h5ad file. Relative paths are resolved against the MCP server's working directory, which may not match the user's.
/evaluate-h5ad identifies problems. /curate-h5ad applies the safe, mechanical fixes and hands back a punch list of everything still needing a curator's decision or upstream data.
The target schemas are:
cellannotation_schema_version, cellannotation_metadata)description, ambient_count_correction, doublet_detection, default_embedding.author_cell_type) need curator-reviewed mappings, not a blanket NaN conversion.Start with the evaluator, then gate the HCA validator on the schema it reports:
`get_storage_info.encodings` is informational since hca-validation-tools#641 — nothing refuses these encodings. If unsupported_count is greater than 0, the named elements use a nullable-string encoding, and every write normalizes what it touches: full-rewrite tools (whatever streams through write_h5ad — compress_h5ad, normalize_raw, populate_labels, ...) normalize every such element and report the locations under encodings_normalized; in-place tools (rename, merge, backfill) normalize the elements they rewrite as part of the same write; convert_cellxgene_to_hca normalizes its whole output. Just note the count and proceed with the fix sequence — the flags clear as writes happen. Two exceptions. Masked (null) string values are a hard stop: a tool that meets one refuses by name, because they have no plain representation — raise it with the wrangler rather than flattening it. An unopenable file no longer shows up here at all: since hca-validation-tools#661 every tool opens through anndata first, so get_storage_info on a file anndata cannot read — masked categories, a truncated download — returns {"error": ...} carrying anndata's own message, never an encodings map. Treat that error as the verdict: the file is out of scope for these tools and the fix belongs upstream with the producer (docs/anndata-tools-contract.md, Scope). We are not in the diagnosis business, so do not reach for another tool to find out why — none of them will answer either. No tool leaves a partial output — an edit either fully applies or leaves the file untouched. That is not the same as a usable one: a complete snapshot of an out-of-scope file is still unopenable, so a clean run there is not a clean verdict.
Separately, stop if index_masked is a number greater than 0: the index contains nulls, which corrupt joins silently, and that is a problem with the data rather than with our tools. index_masked is null — not 0 — when the index carries no mask of its own, which is the ordinary case; on its own null is not a finding. A categorical index is the exception, and it cuts the other way: its nulls are codes of -1 over plain categories, which _mask_count cannot see and unsupported does not list — the report comes back clean on a file whose cell IDs contain nulls. On a categorical index null means not checked; only a tool that reads the index through read_index will refuse it (#659).
/evaluate-h5ad $ARGUMENTS — produces the structured overview report (schema type, X verdict, matrix & embedding gate, metadata, storage, embeddings, CAP, edit history, summary). This already calls check_schema_type and check_x_normalization, so their verdicts are available for Step 2 gating without a separate tool call.schema: "hca", run validate_schema $ARGUMENTS with object_kind — the HCA schema validator (is_valid, full errors and warnings lists). object_kind is required (#770): pass "source" for a source dataset or "integrated" for an integrated object, as the tracker lists the file. It is never read off the path — if the wrangler has not said which it is, ask; never guess. These are the authoritative blocking/advisory signals for Bucket B decisions. Feature-ID warnings are ordered last; summarize repeated shapes in the punch list rather than pasting thousands of lines verbatim. Since #728 a classified gene ID has no warning of its own: it has one row in the Details: block (old -> new what happened [action], grouped by class), and the Summary: block above it gives the counts. What still prints as individual Feature ID '…' not found lines is features the check cannot classify — transgenes, custom features — and those are the only ones to read one by one. That rule does not extend to `gene_annotation_version` warnings. They are gene-shaped, so they read like part of the feature-ID wall, but they are not in it: the validator sorts feature-ID warnings last, so these appear before it. A file declaring several versions gets one per value, plus assembly, unknown-identifier and no-release findings; a file declaring none -- no column, or only placeholders -- still gets the unknown-identifier and no-release findings where they apply, plus one "is not declared" line -- unless it has no human ENSG ids and no cell is stated human (a mouse file, or no organism column), which stays silent by design; so there is no small fixed number of them -- but each carries the numbers that are the finding — the declared value, how many genes contradict it, named examples, and the releases that do explain the file. Quote them verbatim; summarizing one away destroys it.object_kind you pass to validate_schema; #776). Run find_source_datasets $ARGUMENTS, then compare it with list_source_datasets for the atlas version being curated, with generation passed explicitly — never the default, because 1.1 and 2.0 can be in progress at once; if the wrangler has not said which version, that is a B1 question. Report all of it:source_dataset.status: absent, correct (every cell compared and right), incomplete (some cells empty), unverifiable (some cells could not be compared, because no single source claims them — partition says why) or wrong, with the agree / unset / wrong / unverifiable counts, and for wrong every wrong_values row (the value, what it should be, its cells).partition. On "incomplete", its findings name the cells no source or more than one source claims; the column cannot be set..h5ad removed: the tracker's file name carries it, and an integrated_objects entry may. For each source with matched_by_id above zero, two stop the write and send it to Bucket B with the discrepancy instead of Bucket A:<tracker_name>.h5ad is not a listed file name — the file was most likely renamed on the tracker, or the local copy belongs to another atlas version; get the current copy and run the check again.matched_by_id is above that file's cell_count.Two are listed but do not stop:
matched_by_id below cell_count — normal, since not every source cell is integrated; list each with both counts.integrated_objects disagreeing with the match, checked for every candidate, zero-match rows included — a source with cells whose list lacks this object's tracker_name, or one whose list names this object while no cell matches it. The cell-ID match is the truth; the tracker's links are known to be wrong in places (liver's are inverted).schema: "cellxgene", do not run validate_schema yet — the HCA validator would report a large, mostly irrelevant error list. convert_cellxgene_to_hca moves into Bucket A; after it runs, re-enter Step 1 on the converted file to get the accurate HCA findings.Neither validator finds privacy-sensitive columns under non-canonical names. The schema forbids exactly two obs columns by name — self_reported_ethnicity and self_reported_ethnicity_ontology_term_id, each carrying requirement_level: forbidden — and strip_forbidden_obs_columns removes those same two literal names. Anything carrying the same information under another name passes both untouched, so a file can reach is_valid: true still holding self-reported ethnicity, which HCA forbids for privacy.
No pattern finds these. Across the breast-v1 source datasets the same data appears as ethnicity_verbatim, ethnicity_grouped, reported_ethnicity, race and self_reported_ethnicity_label — names sharing no prefix, no suffix and no common substring. They are found by reading the column list and recognising what the words mean, which is your job here rather than a tool's.
So: read obs_columns from the evaluator's summary and identify any column that appears to carry ethnicity or race tied to a subject.
Exclude the two canonical names. self_reported_ethnicity and self_reported_ethnicity_ontology_term_id are strip_forbidden_obs_columns's job — Bucket A, and it runs earlier. Do not list them here as well.
The reason is worth knowing, because getting it wrong fails silently. Strip runs earlier and removes both canonical columns, so listing them here again fails the absent-name check — and the tool is all-or-nothing: that failure drops nothing. The non-canonical columns this scan exists for would ship, in a run that otherwise looked like it did its job. This scan covers only the names strip cannot see.
For each remaining candidate, call get_descriptive_stats with attribute="obs" and columns set to just the candidate names — left unset it describes every obs column, which on a file like this is 70-plus columns of noise. Build the case:
| Field | What to report |
|---|---|
| column | the exact name |
| dtype and unique count | dtype always; unique on categorical columns only (see the numeric note below) |
| the values | the category vocabulary, from value_counts=True |
| why | what about the name reads as ethnicity or race |
Show the values. The wrangler is the curator and is authorised to see the file's contents — that is the job — and the unedited source dataset retains the column regardless of what this run drops. Withholding the vocabulary from the curator making the call would only make the decision harder to judge.
Two practical notes, neither about disclosure:
value_counts=True. It returns close to the whole column, which buries the punch list in thousands of lines. Report the unique count and characterise the shape instead — "free-text ethnicity descriptions, most unique to one donor".unique or value_counts, so use view_data if you need to see the codes.Candidates go to B1 — they block, and each needs approve-or-strike from the wrangler individually. Never propose a glob or a name pattern; always enumerate.
Scoped to ethnicity and race, and nothing else. Do not extend it by analogy to other fields that feel sensitive — donor_id, age_*, disease, geography, clinical notes. Those are legitimate HCA metadata, several are required, and proposing them for deletion is out of scope for this skill. HCA forbids self-reported ethnicity specifically; that is the whole of what this scan is for. Note the tool will not save you here: it drops any column you name, schema-named or not (#619) — a clinical field under a producer name (dx_notes, donor_origin_country) drops if you ask.
Flag liberally within that scope. A false positive costs one struck line; a false negative ships ethnicity data.
And say what this does not establish. A column you did not flag is not thereby cleared — this step reduces the risk, it does not eliminate it. An obscurely-named column produces a clean-looking report on a file that still carries the data. That sentence belongs in the saved report, not just this conversation — Step 5 requires it in the Summary, which ships whether or not anything was dropped.
Only these are in Bucket A. Nothing else. A row belongs in A only when its preconditions are already satisfied at punch-list time — don't pre-list rows whose inputs depend on an unanswered B question (e.g. set_uns('default_embedding', …) belongs in B2 until the wrangler picks a value, then gets promoted to A per Step 3).
check_schema_type reports schema: "cellxgene". Must run first: it reshapes the file into HCA layout before any other fix makes sense, and the other tools (including validate_schema) assume HCA layout. Strips the HCA-forbidden SRE columns (self_reported_ethnicity_ontology_term_id, self_reported_ethnicity) as a side-effect — so for CellxGENE-layout inputs, the SRE problem is handled here. After conversion, re-enter Step 1 on the converted file to get an accurate Bucket A/B/C list.obs['self_reported_ethnicity'] or obs['self_reported_ethnicity_ontology_term_id'] is present. Mechanically removes the privacy-forbidden columns and updates obs[column-order]. Required prerequisite for populate_labels (which refuses while SRE is present), so this should run early in the Bucket A sequence on HCA-layout inputs. On CellxGENE-layout inputs the tool refuses — convert_cellxgene_to_hca is the correct entry point there.uns['batch_condition'] names; schema-named columns drop like any other — whether the result is valid is validate_schema's verdict (#619). Do not read that last one as general reference tracing: batch_condition is the only reference the tool knows about, so a column named somewhere else in uns can still be dropped out from under it. It deletes uns['<column>_colors'] alongside each dropped column, since an orphaned palette is a validator error. Note it does not shrink the file: HDF5 doesn't reclaim freed space in place, which is one reason compress_h5ad runs last.check_x_normalization reports verdict: "raw_counts", whether or not has_raw_x is true. Deterministic, and which of two things it does depends on raw.X. With no `raw.X`, it moves X→raw.X and normalizes X with normalize_total(target_sum=10000) + log1p. With a `raw.X` that is the same matrix as X — normalization never ran, so both hold the raw counts — it leaves raw.X untouched and normalizes X alone. A raw.X that differs from X is a genuinely different file and the tool refuses. It also refuses when raw.X does not itself hold counts, and reports a no-op (writing nothing) when raw.X holds counts and X is already normalized, which is the target layout.feature_* columns + 7 obs ontology labels. Missing or all-NaN columns get filled from canonical (term_id → ontology label for obs; Ensembl ID → GENCODE for var) wherever canonical resolves a value. Where it resolves nothing at all, the two cases differ: a missing column refuses rather than writing an empty one (Bucket C), while an existing all-NaN column is reported as matched and left alone. When columns are partially populated, every populated row is verified against canonical, and only NaN rows get filled (the whole column refuses with row-level evidence if any populated row mismatches). Does NOT write observation_joinid (HCA reserved-but-not-required; the value is deterministic per cell index but writing it is label_h5ad's scope, not the populator's). Refuses on obs['self_reported_ethnicity'] or obs['self_reported_ethnicity_ontology_term_id'] present — those two exact names, not a prefix match (run strip_forbidden_obs_columns first) — plus non-human organism, CellxGENE layout (uns['schema_version'] present), or any CellxGENE-derived markers — uns['provenance']['cellxgene'], obs['observation_joinid'], or an import_cellxgene edit-log entry — since add-labels already wrote everything canonically upstream.label_h5ad is not part of this flow — it writes observation_joinid, which makes populate_labels refuse the file from then on. See the labeling decision tree in Step 4 for why.
check_cap_currency reported behind (or absent with a matched export). For the latter source_path is export.path, and behind needs overwrite=True, which strips the old CAP data first in its own edit-log entry; say what moves, e.g. "v4 (2026-08-14, project 1030) → v5 (2026-10-16, project 1070)". Re-run check_cap_currency afterwards: it must report current. Copies annotation sets + cellannotation_schema_version + cellannotation_metadata from the source into the target. Partial overlap is allowed: ≤5% of cells may be missing on either side; HCA rows absent from CAP get NaN in the new CAP columns. If either side exceeds 5% missing the tool aborts — treat that as a Bucket B item and bring it back to the wrangler (usually the CAP source is stale or wrong). The gate is computed from the raw counts — missing_from_hca.n / n_cap and missing_from_cap.n / n_hca, note the crossed denominators — not from the cells.missing_from_*.pct the result reports: that pct is rounded to 1 dp, so a file at 5.04% reports 5.0 and still aborts. Gene-axis overlap is recorded but does not gate the copy: a CAP source with extra genes is fine.partition: "exact", no stopping discrepancy, and an existing column that is not correct (absent, incomplete or wrong; unverifiable cannot occur on an exact partition). Say which, with the counts — e.g. "wrong: 10,389 cells hold Burclaff2022, expected hgca-v1-burclaff-2022". The column is the tool's own, so it replaces whatever is there, keeping its position; it recomputes the match itself rather than trusting the preview. Report its per-source cells. A correct column needs nothing — say "already set and correct" (#776).get_storage_info shows no HDF5 filter on X's underlying dataset (X.data.compression for sparse X, X.compression for dense X). If the file is already compressed, the tool safely returns {skipped: true, reason: ...} rather than rewriting. Pure compression, no data change.Split these into two classes so the wrangler sees which items actually block validation vs. which are recommended-but-optional. The primary blocking signal is validate_schema — any error it reports (on obs, var, or uns) blocks. Use list_uns_fields as a secondary signal for missing uns fields specifically: required: true fields that are unset are blocking; required: false fields that are unset are recommended at most.
The privacy scan is the one B1 source that does not come from a validator. A file carrying ethnicity under a producer name reaches is_valid: true — that is the whole defect the scan exists for — so those items block on HCA policy while every validator signal reads clean. Do not downgrade them to B2 on the grounds that nothing flagged them.
For each item, write a concrete question. For B1 items, do not include a suggested answer — ask only for the missing required value. For B2 items, if there's an obvious single valid option (e.g. only one 2D embedding exists), you may phrase it as a confirmation question ("X_umap — confirm?") rather than silently deciding.
B1 — Blocking (validator errors, unset `required: true` fields, or a privacy finding)
set_source_dataset applies and the wrangler has not said — its check needs it, and it is never left to the default.uns fields (e.g. study_pi) — ask for the value(s).drop_obs_columns call in Step 4 with those names enumerated. Before that call, restate the exact column list and require an explicit yes to that list. A general "drop the ones you flagged" is not naming — echo the names back and wait. Silence is never approval. If the wrangler declines a column, record that in the report — a deliberate keep and an unnoticed column should not look the same to the next reader.copy_cap_annotations reads the source via AnnData/h5py so a URL must be downloaded locally first. If supplied, copy_cap_annotations becomes a mechanical fix for Step 4.check_cap_currency reported export_stale: no local export carries the tracker's cap_url, or the one that does is older than the object's copy. Ask the wrangler to download the current export from that cap_url into cap-source/; then evaluate again, and the re-import becomes Bucket A.uns field the validator flags as missing.B2 — Recommended (optional fields the wrangler may want to set)
Only the fields explicitly named below belong in B2. Do not scan list_uns_fields for other unset optional fields and invent questions about them — a field being optional-and-unset is not itself a reason to ask. The skill's scope is the explicit tool list (convert_cellxgene_to_hca, strip_forbidden_obs_columns, drop_obs_columns, normalize_raw, replace_placeholder_values, populate_labels, copy_cap_annotations, set_uns on the named fields here, compress_h5ad); everything else is the wrangler's call, unprompted.
default_embedding — list the obsm keys and ask which one. Optional per schema, but a file shipped without it will display in CELLxGENE Explorer with no default scatter. Must name a 2D embedding to actually plot; 30D latents (e.g. X_scVI) are valid per schema but won't display. If only one 2D embedding exists, surface that — the wrangler will almost certainly pick it.If the wrangler answers a B2 item during the session, that answer becomes a set_uns mechanical fix (promoted to Bucket A) for Step 4.
Report these but don't attempt to fix:
library_id) — needs real values from source.check_source_obs on an integrated object, from /evaluate-h5ad Section 3) — every metadata field that differs, or is set on one side only, goes here with its counts and top value pairs. Never fix one by copying either side's values: the tool does not say which side is right, and in kidney v1 it is the object for library_id and the source dataset for donors HKB26/HKB27. Cell-type rows are listed as information, not as items.ambient_count_correction / doublet_detection obs columns — per-cell values must come from the upstream source (each source dataset's processing record). Do not broadcast a single value. Report fill rate and move on.library_preparation_batch containing "lib1; lib2; lib3") — needs per-cell resolution, not placeholder replacement.Summary:/Actions:/Details: blocks, and the only thing that says what to do next. Route each class:| tag | whose it is | what to do |
|---|---|---|
[rename] | producer | relay: plain renames, safe to apply in place. The line names the successor |
[review] | producer | relay as a decision, never as a fix. Ensembl has made several of this file's columns one gene; whether their counts can be added is a judgement about the data. Do not remap these |
[drop] | producer | relay: nothing in the allowed gene set to point at, so the feature can only go. Patch/alt genes (not on primary assembly) land here too, and in quantity they are a finding of their own — the file was aligned against a reference including patch sequences, which makes its counts hard to compare |
[drop or re-align] | producer | relay: a split locus cannot have its reads divided after the fact, but they are recoverable by re-quantifying from source |
[strip suffix] | producer | relay: remove the version suffix. The row's status says whether that is the whole problem: version suffix alone means the bare gene is in the allowed set; issued after the allowed set; version suffix means it is not, and the bare gene then reads as [none] above |
[none] | nobody — do not relay as a defect to the producer | the gene is still listed in Ensembl, issued after the allowed gene set; the file's annotation is newer than ours. Note it, and flag that the vendored cellxgene-schema is behind |
[ask] | producer, as a question | ask which genome annotation the file was built against; these identifiers come from outside Ensembl's GRCh38 history |
; disjoint on a Details row | ours first | the old and new spans share no positions, or sit on opposite strands (19 of the table's 39 are strand flips of one locus). A flag is geometry, not a verdict: look at the pair before saying anything, and relay only if the pair really names two different loci |
; old contains new or ; overlap on a Details row | nobody — do not relay as a defect to the producer | Ensembl redrew the gene's extent as well as renaming it. The replacement stands. Worth a sentence only if the producer is combining those features |
| "gene IDs are not in GENCODE v…" | — | the headline; quote its counts, do not relay it on its own |
validate_schema warning, not a tool you can call; #710) — these are not one finding, and they do not all belong to the producer. Route each by its wording before writing anything:| the warning says | whose it is | what to do |
|---|---|---|
| genes "did not exist in" / "are not in" the declared release | producer | relay: the declared value and the gene list disagree. On an integrated object, compare each value with what its source dataset declares. Relay a mismatch; otherwise relay the source dataset's own finding |
| "is not declared" | producer, as a question | ask which annotation their pipeline used. The window it names is evidence to check that answer against, not a value to fill in (see below). "cannot be inferred" or "no human Ensembl gene ids to infer it from" is the same question with no evidence attached; the no-release or unknown-identifier finding beside it is routed by its own row |
| "name different assemblies" | producer | relay: gene_annotation_version and reference_genome contradict each other |
| "is a GRCh37 annotation ... this reference data does not cover" | ours | nothing to relay. Nothing says the declaration is wrong — only that our table starts at r76, so a genuinely GRCh37-aligned file cannot be checked. Extending it is #724 |
| "did not exist until after Ensembl r76 ... cannot have been in rN" | producer | relay: the declared release is below the reference data's range, but the file's own genes rule it out anyway. On an integrated object, compare each value with what its source dataset declares. Relay a mismatch; otherwise relay the source dataset's own finding |
| "older than this reference data covers" | ours | nothing to relay. The declared release predates the gene table; extending it is #724 |
| "predates both" | producer | relay: the declared release is older than GRCh37 (r54 and earlier used NCBI36), so no assembly is claimed and the genes cannot be checked against it |
| "names a genome assembly rather than a gene annotation" | nobody — do not relay as an error | the schema's own documented example is an accession (#719). The producer followed the documentation. Note it and move on |
| "could be Ensembl rN or GENCODE N" | producer, as a question | ask which scheme they meant; do not guess |
| "are in none of the releases this reference data covers" | producer | relay: the gene list points at a reference we do not have |
| "No Ensembl release … contains every one of this file's known genes" | by its last sentence | "This file is an integrated object, so this is expected" — nothing to relay: a union of sources matches none of them, and what each value must match is the source dataset its cells came from. "This file is a source dataset, so its gene list spans releases" — producer; relay |
| "newer than this reference data covers" | ours | nothing to relay. Our gene table needs regenerating — see the package README |
The findings that rule out a declared release (too early or too late) and the no-release findings end in one sentence chosen by the object_kind you passed to validate_schema — "This file is a source dataset …" or "This file is an integrated object …" (#774). It restates what you told the validator, not something read off the file. If it disagrees with how the tracker lists the file, the call was wrong: re-run with the right kind before routing anything.
Nothing here is fixed by a tool. set_uns cannot reach an obs column, and backfill_obs_from_source fills only gaps — it never overwrites a value that is already set, reporting a disagreement as a conflict and writing nothing. These findings are by definition values that are present and wrong, which is exactly the case backfill declines.
Do not write the validator's suggested releases back into the file. The message names the releases consistent with the gene list — "This file's gene set is consistent with r105 to r110" — and that looks like the answer. It is not. It is a range, not a release; and because most real files are filtered to detected genes, the range is usually a superset of the truth, so the real annotation may be narrower. Treat it as evidence for the producer to confirm against their own pipeline record, never as a value to fill in. Rule 1 applies with force here precisely because the validator hands you a specific-looking number.
Several distinct values on an integrated object is not by itself a defect. The check reports one finding per declared value, and an integrated object legitimately spans its sources' annotations — its var is a union of its sources. A single release may still fit that union, and the check will say so when one does — what each declared value must match is the source dataset its cells came from, which the check cannot confirm without that linkage (#719). Do not present the multiplicity to a producer as something to fix. What the grain should be is unsettled (#719): the object has one var index, so one annotation describes its feature space, but the column is per-cell and may be recording each source dataset's annotation. Report the findings you see and leave that question to #719.
author_cell_type variants — needs a curator mapping.copy_cap_annotations runs mechanically.)uns['schema_version'] — signals CellxGENE layout, i.e. it has already been through cellxgene-schema add-labels. populate_labels refuses to run (it checks that key specifically, not schema_reference). The path is convert_cellxgene_to_hca (Bucket A, runs first), not a request to upstream. Do not strip the key by hand.obs['organism_ontology_term_id'] value other than NCBITaxon:9606 — populate_labels is human-only. Supporting another organism is a code change, not a curation fix.*_ontology_term_id (or with GENCODE, for the var feature_* columns). A pre-populated column is not by itself a blocker, but a conflicting one blocks the whole run: populate_labels classifies every column first and writes nothing at all if any of them mismatches, so one bad column also withholds the fills that would have succeeded. Its report groups each disagreement by (term ID, file label) with row counts, e.g.:obs['tissue']: 4812 rows labeled 'lung' but tissue_ontology_term_id is 'UBERON:0002097' (canonical label: 'skin of body'). Either fix tissue_ontology_term_id to match the label, or fix the label to match tissue_ontology_term_id.It deliberately does not choose a side. A mismatch means the label and the term ID disagree, and only someone who knows the dataset can say whether the producer mislabeled the cells or picked the wrong ontology term — guessing wrong corrupts real biological metadata. Bring the tool's verbatim message to the curator; every conflicting column has to be resolved before any labeling lands.
*_ontology_term_id source column is absent — e.g. free-text obs['cell_type'] with no cell_type_ontology_term_id, which the schema permits since that term-id column is optional. populate_labels refuses that column (it can neither verify nor fill without the source), and per the item above that blocks the whole run. No mismatch is involved, so this reads differently from a disagreement: upstream needs to supply the term IDs, or the free-text column has to go (drop_obs_columns removes producer columns the schema doesn't name). The same applies to the raw.var feature_* mirrors, which are classified independently and can refuse on their own.*_ontology_term_id source resolves no label for any row — the source is either entirely empty, or none of its values are terms the ontology recognizes. populate_labels refuses rather than writing an all-NaN label column, and per the item above that blocks the whole run. The same shape on the var side is a var.index that holds gene symbols rather than Ensembl IDs, so GENCODE resolves nothing. Both are upstream data problems, and both are independently hard validate_schema errors ("must not contain NaN values", "is not a valid ontology term id"), so expect the validator to name the column too — quote its error when asking upstream for the fix.uns['cellannotation_metadata'] or other CAP-related uns keys that aren't structurally a dict (e.g. legacy non-dict serializations) — the cell-annotation validator catches these; report verbatim and bring to the CAP curator. (Note: the previous-generation issue here — SRE columns on HCA-layout inputs — now has a Bucket A path via strip_forbidden_obs_columns. CellxGENE-layout inputs are still handled by convert_cellxgene_to_hca as a side-effect.)uns keys instead of nested under uns['cap_metadata']. drop_obs_columns refuses such a file outright, whatever columns were named, and copy_cap_annotations refuses it as a source; the refusal fires even on a mixed-layout file that also carries a nested block. Not fixable here: the CAP file has to be re-exported with its metadata nested. Report the tool's verbatim message and bring it to the CAP curator.validate_cell_annotation error other than NO_SETS_ERROR (e.g. missing required --<suffix> obs columns on an annotation set, malformed cellannotation_schema_version, per-set metadata not a dict). These are CAP-side structural defects — copy_cap_annotations faithfully copies what the CAP source provides, so the fix has to land in the CAP export. Surface the validator's verbatim error so the curator can ask the CAP team for a corrected export. (The NO_SETS_ERROR case is the B1 "no CAP source provided yet" workflow above.)Show these sections: A (will run), B1 (blocking — needs your answer), B2 (recommended — optional), C (still to do, out of scope). Then stop and wait for explicit approval before running anything.
If the wrangler answers any Bucket B items (B1 or B2), promote those to Bucket A as the appropriate mechanical action: set_uns for answered uns values (e.g. default_embedding, study_pi), copy_cap_annotations when the answer is a CAP source file path, drop_obs_columns for privacy columns approved by name.
Order:
convert_cellxgene_to_hca first if applicable — then stop, re-run Steps 1–3 on the converted file before continuing (conversion changes the layout enough that the prior punch list is stale).strip_forbidden_obs_columns next if applicable (HCA-layout input with SRE columns present) — must run before populate_labels, which refuses while SRE is present. On CellxGENE-layout inputs this is unnecessary; the convert step above already stripped them.drop_obs_columns if the wrangler approved any privacy columns in Step 3 — one call, approved names enumerated. This step is independent of step 2: a file with no canonical SRE columns skips that step and still reaches this one, which is the normal shape for the producer-named columns this targets. Removal belongs ahead of the content edits so labeling and CAP see the column set the file actually ships. It does not shrink the file; compress_h5ad at the end repacks.Every refusal leaves the file untouched — that is the tool working, not a failure to route around. Its checks on the names themselves (the obs index, a name that isn't present) are listed with the tool in Bucket A; three more can fire at this step:
uns['batch_condition'] names one of the columns → resolving that is a Bucket C decision for the wrangler, not a reason to drop fewer columns.-- while uns['cap_metadata'] is present → that is a CAP annotation-set column; bring it to the CAP curator.Report the refusal verbatim and re-plan; never retry with a trimmed list to get past it.
normalize_raw, each replace_placeholder_values, labeling step (decision tree below), copy_cap_annotations (if a source was supplied, or check_cap_currency found the object behind), any set_uns approved in Step 3, and set_source_dataset (integrated objects, when it is in Bucket A and the wrangler approved it in Step 3). Whichever labeling tool ran (if any) must precede copy_cap_annotations — copy_cap_annotations calls validate_marker_genes, which reads var['feature_name']; populating it first gives marker-gene validation canonical gene symbols to match against.Labeling decision tree — exactly one branch fires (or zero, when no labeling is needed):
uns['schema_version'] present, uns['provenance']['cellxgene'] present, obs['observation_joinid'] present, import_cellxgene in edit log) → skip labeling. The file's controlled label columns were already populated canonically by cellxgene-schema add-labels upstream and preserved through any conversion, and populate_labels correctly refuses on these files.*_ontology_term_id source carries values, verifies a partly-filled column row by row before filling only its NaN rows, and skips an obs label column when that column and its *_ontology_term_id source are both absent. Two cases are errors rather than skips, and one error withholds the whole run (see the corresponding Bucket C items): a label column present without its source, and an absent label column whose source resolves to no label for any row (all-NaN, or term IDs the ontology doesn't recognize). It never writes an all-NaN column.If neither branch fits cleanly (e.g. file has SRE present), populate_labels surfaces a refusal pointing at the prerequisite (strip_forbidden_obs_columns etc.) — handle that first and re-classify.
label_h5ad is deliberately absent from this tree. It fills the same labels, but also writes obs['observation_joinid'] — a column HCA has no use for, and whose presence makes populate_labels refuse the file from then on. Labeling with it is therefore a one-way door: the labels can never be refreshed after an ontology update. It also refuses outright if any controlled column is already populated, where populate_labels verifies those rows and fills the rest.
Note it is not a route to a CellxGENE-ready file either: HCALabeler applies the HCA schema and deliberately skips the CellxGENE uns writes (schema_version, schema_reference, organism), and its preflight refuses a file that already carries schema_version or schema_reference — the only two uns keys it inspects. An incoming uns['organism'] is skipped on write and is not itself a refusal; obs['organism'] is a different matter — the labeler refuses any add-labels target column that is merely present, whether or not it holds values. Do not reach for it on the assumption that it serves a CellxGENE handoff.
compress_h5ad last.Each tool writes a new timestamped file. For most subsequent calls, passing either the original path or the latest works — resolve_latest picks up the newest variant automatically. Two exceptions: convert_cellxgene_to_hca does not auto-resolve (call it with the exact path you want to convert), and copy_cap_annotations only auto-resolves its target_path (the source_path is used verbatim).
Re-run view_edit_log on the final file, then produce a structured report with these sections in order. Also re-run validate_schema — but only if check_schema_type reports hca on the final file. If the file is still CellxGENE (e.g. conversion wasn't approved), skip the validator rerun and note why under "Validator delta" instead of pasting a misleading error list. Use markdown tables; skip any section with no content.
Also re-run get_cap_annotations on the final file, and if it reports has_cap_annotations: true, run validate_cell_annotation on the final file too. This is the structural validator the dataset-validator service runs at upload time under the hcaCellAnnotation key — running it here catches issues during curation instead of post-upload. Gate on has_cap_annotations directly (not on the edit log) so the validator runs on any file that ships with CAP, including files where the producer pre-attached CAP without going through copy_cap_annotations. Its output feeds the Validator delta section's cell-annotation rows.
For the Provenance line below, re-run get_summary on the final file to fetch its obs columns, then run get_descriptive_stats with columns set to the intersection of ["donor_id", "sample_id", "library_id"] and the final file's obs column names (extract name from each {name, dtype} object in get_summary.obs_columns).
Two or three sentences distilling the session: which Bucket A operations actually ran, the validator delta in one phrase (e.g. "errors went 4 → 2; remaining errors are Bucket C upstream-data issues"), and a one-clause hand-off (e.g. "Bucket B1 awaiting wrangler input on study_pi"). Tight prose paragraph, no nested headings.
End the paragraph with the privacy-scan line, always, including when the scan found nothing and no column was dropped: "A privacy scan for ethnicity and race columns was run; columns not flagged by it are not thereby cleared — it reduces privacy risk, it does not eliminate it." A clean file is exactly the case a reader is most likely to mistake for a clearance, and it is the case where every other trace of the scan is omitted: the report's drop_obs_columns row only appears when that tool ran. This sentence is the one that always ships.
Then add an Outstanding issues bullet list pulled from Buckets B1, B2, and C — one line per item, no bucket-label prefixes (the reader doesn't need our internal taxonomy). Order: validator errors first, then warnings, then non-validator items. The full detail with action questions lives in the Still to do section near the bottom — keep this list tight: one line per item, no inline tables. If all three buckets are empty, replace the bullet list with a single line: "Outstanding issues: none."
One short paragraph or bullet block with: final file path, shape (n_obs × n_vars), title from uns, schema type (include version only when schema is CellxGENE — HCA is unversioned), X verdict + raw.X presence, compression status, obsm keys present. Add a Provenance line: N donors · M samples · K libraries from get_descriptive_stats.columns[<col>].unique for each column. Skip any metric whose column wasn't returned or whose unique is 0.
| Operation | Effect |
|---|---|
drop_obs_columns | Name every column dropped, from the tool's own obs_columns_dropped, and say why — e.g. "Dropped ethnicity_verbatim, ethnicity_grouped, self_reported_ethnicity_label — privacy-sensitive ethnicity data under non-canonical names, approved individually." Name the columns; there is no need to restate their values, which the punch list already covered. If the wrangler declined a candidate, record that too, so a deliberate keep is distinguishable from a column nobody looked at. (The "not thereby cleared" caveat lives in the Summary, which ships whether or not this row does.) |
normalize_raw | e.g. "Moved raw counts → raw.X; normalized X with normalize_total(target_sum=10000) + log1p", or "raw.X already held the same counts and was left unmodified; normalized X with normalize_total(target_sum=10000) + log1p". The tool's raw_x field says which. |
replace_placeholder_values (library_preparation_batch) | e.g. "N cells: 'unknown' → NaN" |
populate_labels | Name the columns from the tool's own filled and matched lists — e.g. "Filled var['feature_name'], feature_reference, cell_type, tissue; assay and sex already matched". The tool returns column names, not row counts, so do not quote per-row figures here unless another tool supplied them. Populated rows are verified against canonical before anything is written, so this step never overwrites producer text. |
copy_cap_annotations | name the CAP source file |
compress_h5ad | e.g. "Skipped — already gzipped" or "Rewrote X with gzip level 4" |
Only include the rows for tools that actually ran this session.
| Before | After | |
|---|---|---|
hcaSchema errors | N | M |
hcaSchema non-feature-ID warnings | N | M |
hcaSchema CAP zero-observation warnings | N | M |
hcaCellAnnotation errors | N | M |
hcaCellAnnotation warnings | N | M |
| Named issues resolved | — | e.g. "raw.X absent", "unknown placeholder in library_preparation_batch" |
gene_annotation_version warnings land in the non-feature-ID row and are expected to be identical before and after — nothing in Bucket A touches that column, and nothing should. A persistent count there is the check working, not a fix that failed, the same way the CAP zero-observation row persists by design.
Counts mirror the two validators the dataset-validator service runs at upload time: hcaSchema (Tier 1 + cosmetic checks, from validate_schema) and hcaCellAnnotation (CAP structural checks, from validate_cell_annotation). Each validator's rows (all of its hcaSchema:* rows, or all of its hcaCellAnnotation:* rows) omit cleanly together when that validator wasn't run this session — e.g. the cell-annotation rows are skipped wholesale on files without CAP.
Count CAP "zero observations" warnings (text: contains a category '...' with zero observations) separately from other hcaSchema warnings. These are expected after copy_cap_annotations: CAP declares a closed vocabulary per annotation set that spans all lineages, and a per-lineage file only realizes a subset — unused vocabulary terms are intentional schema information, not a defect. Report the count and move on; don't prune them. The validator's --add-labels remediation note comes from vendored CellxGENE code and does not apply to HCA.
Also list the specific error/warning kinds that disappeared or newly appeared, one line each, prefixed with their validator (hcaSchema: or hcaCellAnnotation:).
copy_cap_annotations ran this session, or a prior import_cap_annotations entry is in the edit log)Pull from the latest import_cap_annotations entry's details:
| Metric | Value |
|---|---|
| CAP source file | cap_source_file |
cells.n_cap | … |
cells.n_hca | … |
cells.n_matched | … |
cells.missing_from_hca | n (pct% of CAP) |
cells.missing_from_cap | n (pct% of HCA) |
genes.n_cap | … |
genes.n_hca | … |
genes.n_matched | … |
genes.missing_from_hca | n (pct% of CAP) |
genes.missing_from_cap | n (pct% of HCA) |
copy_cap_annotations ran this session, or a prior import_cap_annotations entry is in the edit log)Source the numbers from the copy_cap_annotations tool result's marker_gene_validation field if it ran this session. If only a prior import_cap_annotations entry exists, call validate_marker_genes on the final file to get fresh numbers — a marker list that matched against var.index before populate_labels filled var['feature_name'] will look very different now.
Marker symbols are resolved against the target's var gene-name source: var['feature_name'] is preferred, else var['gene_name'], else var.index (the Ensembl IDs) as a last resort. Files that went through populate_labels have feature_name filled wherever GENCODE knew the Ensembl ID; files that skipped labeling fall back to whatever the producer shipped.
| Metric | Value |
|---|---|
| Total unique markers | … |
| Found in var gene-name source | … |
| Missing | … |
For each missing marker, list it with its classification exactly as returned by the tool — not_in_gencode (marker symbol doesn't resolve to any GENCODE entry — typo, glob pattern, or deprecated rename), missing_from_var (valid symbol but not present in this file's gene set), or known_rename (the submitted marker is a valid current GENCODE symbol, but this file's var carries the gene under a different — usually deprecated — name, given in var_name, with the matching Ensembl ID in ensembl_id):
| Marker | Classification | Var name | Ensembl ID |
|---|---|---|---|
| … | … | … | … |
Leave Var name / Ensembl ID blank for not_in_gencode and missing_from_var rows — those fields are only populated on known_rename. If all markers hit, say so in one line instead of an empty table.
Report each missing marker by symbol and classification only. Do not speculate about why a symbol is not_in_gencode (e.g. "looks like a glob", "looks like a truncated STMN1", "probably a typo") — the classification name is the answer the tool gave; cause is the curator's call. Pointer for follow-up: not_in_gencode and missing_from_var go to the CAP curator (CAP-side fix); known_rename is a file-side naming difference, not a CAP defect — the CAP symbol is already current, and var_name is the older name this file uses for the same gene.
Bucket B1 — blocking (validator errors, unset `required: true` fields, or a privacy finding)
| Field | Question |
|---|---|
study_pi | who are the PI(s)? e.g. ["Teichmann,Sarah,A."] |
Bucket B2 — recommended (optional)
| Field | Question |
|---|---|
default_embedding | X_umap (only 2D option) — confirm? |
Bucket C — upstream / curator
| Issue | Detail |
|---|---|
library_id NaN (validator error) | Needs real values from source |
gene_annotation_version contradicted by the genes | Declared v87; 80 of 20,032 genes postdate it. Releases r105–r110 explain the file — a window, and a superset for a filtered gene list, so the producer confirms against their pipeline record rather than us filling it in |
Only surface items that are still open — don't re-list anything resolved this session. Omit any of the three sub-tables that have no entries.
Render every entry returned by the view_edit_log call from the top of Step 5 as a table, oldest first:
| # | Timestamp (UTC) | Operation | Description |
|---|---|---|---|
| 1 | 2026-04-21 04:19:10 | normalize_raw | Moved raw counts to raw.X and normalized X with normalize_total(target_sum=10000) + log1p |
Format the timestamp as YYYY-MM-DD HH:MM:SS (drop the T and the fractional seconds and timezone — entries are always UTC). Use the entry's description field verbatim. If the file has no edit log (no uns/provenance/edit_history), say "No edit history — file hasn't been edited through hca-anndata-tools."
After rendering the full session report on screen, use the Write tool to save the same markdown to a file alongside the h5ad. Path: same directory as the input file, basename of the input minus the .h5ad extension, then -curation-report-<YYYY-MM-DD>.md (use today's date). Example: /foo/bar/myeloid.h5ad → /foo/bar/myeloid-curation-report-2026-05-07.md. Overwrite if it already exists. After saving, confirm the path back to the user as a single line.
~30 seconds. Free. No account. Every finding cites a rule and a line of evidence.