Data Quality Report

A governance-grade reporting skill. Produces auditable PDF deliverables aligned to DAMA-DMBOK2 standards, with traceable lineage and compliance checks.

When to use this skill

Activate when the user has a dataset and needs a formal, archivable quality report — not just an exploratory analysis. Typical signals:

• "Generate an audit-ready PDF for this dataset"
• "I need a quality report I can send to compliance / regulator / management"
• "Produce a DAMA-aligned data quality assessment"
• "Génère-moi un rapport qualité que je peux archiver"
• "Audit cette base de données pour la conformité"

Pre-condition: the dataset should be loaded and reasonably structured. If raw quality is very low (>50% missing across many columns), call eda-explorer first to surface issues, then loop back to this skill once the user decides on a cleaning strategy.

Do NOT activate this skill for:

• Quick exploration → use eda-explorer
• Feature preparation → use feature-engineer
• Real-time monitoring (this is a snapshot skill, not a streaming one)

Workflow

For every request, follow these 6 phases in order. Never skip a phase.

Phase 1 — Profiling

Run a complete profile of the dataset. Reuse the logic from eda-explorer if it produces results in the same conversation; otherwise, compute fresh:

• Shape, dtypes, memory
• Missing value counts and percentages per column
• Cardinality and uniqueness per column
• Numeric statistics (min, max, mean, median, std, IQR)
• Outlier counts (IQR + Z-score methods)
• Datetime range for date/time columns
• PII detection (see Phase 4)

Store all metrics as a Python dict — they will be reused in Phases 2-6.

Phase 2 — DAMA-DMBOK 6 dimensions mapping

Compute a score from 0 to 100 for each of the six dimensions:

Timeliness rule (heuristic):

• If no datetime column exists → None with note "No datetime column to assess"
• If max datetime is within 30 days of today → 100
• If within 90 days → 80
• If within 365 days → 60
• Older → 40
• This is a heuristic; the user can override with their own freshness threshold

Aggregate score: weighted average of the 5 computable dimensions:

• Completeness: 0.25
• Uniqueness: 0.15
• Validity: 0.20
• Consistency: 0.15
• Timeliness: 0.25

Color code:

• 80-100 → GREEN
• 60-79 → YELLOW
• 0-59 → RED

Phase 3 — Risk and recommendations

For each dimension scoring < 80, generate 1-3 specific, actionable recommendations referencing actual columns. Categorize as:

• [CLEAN] — fix existing values
• [IMPUTE] — fill missing
• [DROP] — remove column or rows
• [INVESTIGATE] — manual review needed
• [GOVERN] — process / policy change needed

Phase 4 — Compliance checks

Run three compliance modules:

#### 4a) RGPD — PII detection

Scan column names AND sample values (up to 50 rows) for these patterns:

Output: list of flagged columns with category and confidence (high/medium/low).

#### 4b) ISO 8000-110 — Data quality management checklist

Verify these 6 ISO 8000-110 requirements:

#### 4c) BCBS 239 — 14 principles spotlight (banking only)

Only run this section if the user explicitly mentions banking, finance, credit, or BCBS. Otherwise, skip with a note "Skipped — not a banking dataset".

When run, evaluate the 4 most measurable principles:

• Principle 3 — Accuracy & integrity: maps to Validity score
• Principle 4 — Completeness: maps to Completeness score
• Principle 5 — Timeliness: maps to Timeliness score
• Principle 6 — Adaptability: manual assessment

Phase 5 — Generate PDF

Build the PDF using ReportLab (required: pip install reportlab).

Format: A4, portrait, 1.5cm margins.

Color palette (CONEXT brand):

• Navy: #0D0E52
• Teal: #00B5A3
• Light: #F4F5FA
• Ink: #1A1A2E
• Muted: #6B6B85
• Warning red: #C0392B
• Warning yellow: #F2B705

Typography:

• Headings: Helvetica-Bold or Georgia (use Helvetica-Bold for ReportLab simplicity)
• Body: Helvetica, 10.5pt
• Code/values: Courier, 9pt

Mandatory pages in this exact order:

1. Cover page — Title, dataset name, date, generated_by, content hash (sha256 of source CSV first 1MB)
2. Executive summary — Aggregate score (big number, color-coded), 3 key findings, 3 top risks
3. DAMA dimensions detail — One page per dimension scoring < 80, score gauge, sub-metrics, recommendations
4. Schema & profiling — Column-by-column table with type, n_unique, n_null, validity flags
5. Compliance — RGPD section + ISO 8000 checklist + BCBS 239 (if applicable)
6. Recommendations — Grouped by category (CLEAN/IMPUTE/DROP/INVESTIGATE/GOVERN)
7. Audit & signature — Skill version, generated_at, source file path, content hash, signed-off field (manual)

Visual elements:

• Score gauge: horizontal bar with color zones (red 0-59, yellow 60-79, green 80-100), pointer at actual score
• Dimension cards: navy header band, score in big font, recommendations as bullets
• Compliance flags: red icon for direct PII detected, yellow for indirect, green for OK

Phase 6 — Export artifacts

Save 3 files in outputs/quality/:

1. `quality_report_<YYYY-MM-DD_HHMMSS>.pdf` — the human deliverable
2. `quality_report_<YYYY-MM-DD_HHMMSS>.json` — machine-readable, full schema below
3. `audit_log.txt` — append-only log: timestamp + skill_version + source + hash + score

JSON schema (machine-readable export)

{
  "skill_version": "0.1.0",
  "generated_at": "2026-04-27T18:32:11Z",
  "source": {
    "filename": "sample_insurance.csv",
    "shape": [2000, 10],
    "size_bytes": 154280,
    "content_hash_sha256": "ab12cd34...",
    "loaded_at": "2026-04-27T18:32:08Z"
  },
  "dama_scores": {
    "completeness": 96.4,
    "uniqueness": 100.0,
    "validity": 88.2,
    "consistency": 90.0,
    "timeliness": 80.0,
    "accuracy": null,
    "aggregate": 91.2,
    "flag": "GREEN"
  },
  "compliance": {
    "rgpd": {
      "direct_identifiers_found": ["customer_id"],
      "indirect_identifiers_found": [],
      "sensitive_data_found": [],
      "verdict": "ATTENTION — pseudonymization recommended"
    },
    "iso_8000_110": {
      "passed": 4,
      "total": 6,
      "manual_required": ["master_data_registry", "standards_conformance"]
    },
    "bcbs_239": {
      "applicable": false,
      "reason": "Not a banking dataset"
    }
  },
  "recommendations": [
    {"category": "IMPUTE", "column": "satisfaction_score", "action": "..."},
    {"category": "GOVERN", "column": "region", "action": "..."}
  ],
  "audit": {
    "skill_version": "0.1.0",
    "audit_log_path": "outputs/quality/audit_log.txt",
    "pdf_path": "outputs/quality/quality_report_2026-04-27_183211.pdf"
  }
}

Output format (response to user)

After generation, respond with a concise summary:

# Data Quality Report — <filename>

**Score global : XX/100 [GREEN | YELLOW | RED]**

## Synthèse par dimension DAMA-DMBOK2
| Dimension | Score | Statut |
| Completeness | XX | 🟢/🟡/🔴 |
...

## Compliance
- RGPD : <verdict>
- ISO 8000-110 : X/6 vérifiés automatiquement
- BCBS 239 : <applicable ou skipped>

## Top 3 risques identifiés
1. ...
2. ...
3. ...

## Artefacts générés
- 📄 PDF : `outputs/quality/quality_report_<timestamp>.pdf` (X pages, X KB)
- 📋 JSON : `outputs/quality/quality_report_<timestamp>.json`
- 📜 Audit log mis à jour : `outputs/quality/audit_log.txt`

## Prochaines étapes recommandées
1. ...

Constraints

• Always use ReportLab for PDF (no LaTeX, no wkhtmltopdf — keeps the skill self-contained)
• Always generate the JSON alongside the PDF (governance audit requires both)
• Always append to audit_log.txt (never overwrite — append-only is core to auditability)
• Always compute the SHA-256 hash on the FIRST 1 MB of the source file (full hash is overkill for QA, first MB is enough to detect content changes for typical datasets)
• Never include actual PII values in the report — only column names and counts
• Never suggest the user remove the audit log
• Output language: French if the user wrote in French, English otherwise (match user's language)
• PDF font: stick to Helvetica/Helvetica-Bold + Courier (built into ReportLab — no font installation required)

Code skeleton

Use this as the structural backbone. Adapt to actual data, but keep the 6-phase structure intact.

import hashlib
import json
from datetime import datetime, timezone
from pathlib import Path

import pandas as pd
import numpy as np
from reportlab.lib.pagesizes import A4
from reportlab.lib.colors import HexColor
from reportlab.lib.styles import getSampleStyleSheet, ParagraphStyle
from reportlab.lib.units import cm
from reportlab.platypus import (
    SimpleDocTemplate, Paragraph, Spacer, Table, TableStyle,
    PageBreak, Image,
)

# ─── Setup ──────────────────────────────────────────────
SKILL_VERSION = "0.1.0"
OUTPUT_DIR = Path("outputs/quality")
OUTPUT_DIR.mkdir(parents=True, exist_ok=True)

NAVY = HexColor("#0D0E52")
TEAL = HexColor("#00B5A3")
LIGHT = HexColor("#F4F5FA")
INK = HexColor("#1A1A2E")
RED = HexColor("#C0392B")
YELLOW = HexColor("#F2B705")
GREEN = HexColor("#27AE60")

# ─── Phase 1: Profiling ─────────────────────────────────
def profile(df: pd.DataFrame) -> dict:
    return {
        "shape": df.shape,
        "missing_pct": (df.isna().sum() / len(df) * 100).to_dict(),
        "duplicates": int(df.duplicated().sum()),
        "dtypes": {c: str(df[c].dtype) for c in df.columns},
        # ... outliers, datetime range, etc.
    }

# ─── Phase 2: DAMA dimensions ──────────────────────────
def compute_dama_scores(profile: dict, df: pd.DataFrame) -> dict:
    completeness = 100 * (1 - np.mean(list(profile["missing_pct"].values())) / 100)
    uniqueness = 100 * (1 - profile["duplicates"] / profile["shape"][0])
    # ... validity, consistency, timeliness
    aggregate = 0.25*completeness + 0.15*uniqueness + 0.20*validity + 0.15*consistency + 0.25*timeliness
    flag = "GREEN" if aggregate >= 80 else ("YELLOW" if aggregate >= 60 else "RED")
    return {"completeness": completeness, ..., "aggregate": aggregate, "flag": flag}

# ─── Phase 3: Recommendations ──────────────────────────
def build_recommendations(scores: dict, profile: dict) -> list:
    recs = []
    if scores["completeness"] < 80:
        # find the top missing columns
        top_missing = sorted(profile["missing_pct"].items(), key=lambda x: -x[1])[:3]
        for col, pct in top_missing:
            if pct > 30:
                recs.append({"category": "DROP", "column": col, "action": f"...{pct:.1f}% missing — consider dropping"})
            else:
                recs.append({"category": "IMPUTE", "column": col, "action": f"Impute median or mode"})
    # ... validity, consistency
    return recs

# ─── Phase 4: Compliance ───────────────────────────────
PII_PATTERNS = {
    "direct": ["email", "phone", "tel", "name", "nom", "ssn", "nir", "dob", "address"],
    "indirect": ["user_id", "customer_id", "account_id", "ip"],
    "sensitive": ["health", "religion", "politic", "ethnic", "gender"],
}

def check_rgpd(df: pd.DataFrame) -> dict:
    flags = {"direct": [], "indirect": [], "sensitive": []}
    for col in df.columns:
        col_lower = col.lower()
        for category, patterns in PII_PATTERNS.items():
            if any(p in col_lower for p in patterns):
                flags[category].append(col)
                break
    return flags

# ─── Phase 5: PDF generation ────────────────────────────
def hash_first_mb(filepath: Path) -> str:
    with open(filepath, "rb") as f:
        return hashlib.sha256(f.read(1024*1024)).hexdigest()

def generate_pdf(report_data: dict, output_path: Path):
    doc = SimpleDocTemplate(str(output_path), pagesize=A4,
                            topMargin=1.5*cm, bottomMargin=1.5*cm,
                            leftMargin=1.8*cm, rightMargin=1.8*cm)
    story = []
    # Cover page, exec summary, dimensions, compliance, recommendations, audit
    # Build using Paragraph, Spacer, Table, PageBreak
    doc.build(story)

# ─── Phase 6: Export ───────────────────────────────────
def export_artifacts(report_data: dict, source: Path) -> dict:
    timestamp = datetime.now(timezone.utc).strftime("%Y-%m-%d_%H%M%S")
    pdf_path = OUTPUT_DIR / f"quality_report_{timestamp}.pdf"
    json_path = OUTPUT_DIR / f"quality_report_{timestamp}.json"
    log_path = OUTPUT_DIR / "audit_log.txt"

    generate_pdf(report_data, pdf_path)
    json_path.write_text(json.dumps(report_data, indent=2, default=str))
    with open(log_path, "a") as f:
        f.write(f"{timestamp} | v{SKILL_VERSION} | {source} | {report_data['source']['content_hash_sha256'][:12]} | score={report_data['dama_scores']['aggregate']:.1f}\n")

    return {"pdf": pdf_path, "json": json_path, "log": log_path}