Healthcare Data Quality Profiling

Overview

Bad data is the dominant cost driver in US healthcare operations. This skill turns a raw healthcare feed into a quantified, explainable data-quality (DQ) scorecard so that data issues are caught before they corrupt analytics, payment, or ML pipelines. It is built around the six classic DQ dimensions and produces both a human-readable report and a reusable, machine-checkable rule suite (Great Expectations / pandera) that can be wired into CI or a Spark ingestion job.

When to Use This Skill

• Onboarding a new claims, eligibility/membership, provider, or EHR feed and you need to

know whether it can be trusted.

• Standing up an automated data-quality gate that blocks or quarantines bad batches.
• Quantifying the financial/operational impact of "bad data" for a stakeholder.
• Diagnosing why a downstream model degraded (often upstream schema drift or a

completeness cliff).

• Producing audit-ready DQ evidence for a payer, provider, or regulator.

The Six DQ Dimensions (how each is measured)

1. Completeness null / blank / sentinel rate per field; conditional completeness

(e.g., discharge_date required when claim_type = inpatient).

1. Validity values conform to type, range, regex, or code set (NPI is 10 digits and

passes the Luhn check; ICD-10 matches the official code list; gender in allowed set).

1. Consistency cross-field and cross-table logic holds (service_date <= paid_date;

member_age consistent with date_of_birth; sum of line items equals claim total).

1. Uniqueness primary keys are unique; duplicate-record rate (hand off true entity

matching to patient-record-entity-resolution).

1. Conformity formats and units are standardized (dates ISO-8601, currency scale,

provider IDs normalized).

1. Timeliness data freshness / lag vs. SLA; late-arriving partitions.

Workflow

1. Ingest & sample the dataset (CSV/Parquet/DB). For very large feeds, profile a

stratified sample, then promote the rule suite to the full data in Spark.

1. Auto-profile every column (dtype, cardinality, null rate, min/max, top values,

distribution) using the bundled scripts/profile_dataset.py.

1. Apply a healthcare rule pack NPI Luhn validation, ICD-10/CPT/HCPCS membership,

date-order logic, age/DOB consistency, currency sanity, member-ID format.

1. Detect schema drift by diffing the current profile against a stored baseline

(new/removed columns, dtype changes, null-rate shifts, distribution shifts via PSI).

1. Score each dimension 0–100, weight into an overall DQ index, and grade A–F.
2. Emit artifacts: a Markdown/HTML scorecard, a failed_records.parquet quarantine

set with the failing rule per row, and a Great Expectations suite for reuse.

Example

# scripts/profile_dataset.py (excerpt)  pandas + Great Expectations
import pandas as pd, great_expectations as gx

df = pd.read_parquet("claims_2026Q1.parquet")
ctx = gx.get_context()
batch = ctx.sources.pandas_default.read_dataframe(df)

# Healthcare validity rules
batch.expect_column_values_to_match_regex("npi", r"^\d{10}$")
batch.expect_column_values_to_be_in_set("claim_status", ["PAID","DENIED","PENDING","REVERSED"])
batch.expect_column_values_to_not_be_null("member_id")
batch.expect_column_pair_values_a_to_be_greater_than_b("paid_date", "service_date")

results = batch.validate()
score = 100 * results.statistics["successful_expectations"] / results.statistics["evaluated_expectations"]
print(f"Validity score: {score:.1f}")

# Population Stability Index for distribution / schema drift
import numpy as np
def psi(expected, actual, bins=10):
    q = np.quantile(expected, np.linspace(0,1,bins+1))
    e = np.histogram(expected, q)[0]/len(expected) + 1e-6
    a = np.histogram(actual,   q)[0]/len(actual)  + 1e-6
    return np.sum((a-e)*np.log(a/e))     # >0.25 => material drift

Outputs

• dq_scorecard.md / .html per-dimension scores, overall index, grade, top offending

fields, and trend vs. baseline.

• expectations_suite.json reusable Great Expectations / pandera rules.
• failed_records.parquet quarantined rows annotated with the violated rule.
• schema_drift.json added/removed/changed columns and PSI per numeric field.

Healthcare Context

Tuned for payer/provider data: NPI, member/subscriber IDs, ICD-10-CM, CPT/HCPCS, revenue codes, place-of-service, and eligibility spans. Pairs naturally with patient-record-entity-resolution (dedup), claims-anomaly-detection (statistical outliers), and medical-ontology-code-mapping (code normalization). Designed to scale from a laptop sample to a spark-healthcare-data-pipeline job over billions of records.

References

• Great Expectations docs https://docs.greatexpectations.io
• pandera dataframe schemas https://pandera.readthedocs.io
• DAMA-DMBOK data-quality dimensions; CMS data-quality guidance for claims.