EDA Explorer

A reproducible exploratory data analysis skill. Same dataset → same report structure → no surprises.

When to use this skill

Activate this skill whenever the user wants to understand a tabular dataset before doing anything else with it. Typical signals:

• User uploads a .csv, .xlsx, .parquet, or .tsv file
• User asks "what's in this data?", "explore this", "do an EDA", "fais une exploration"
• User mentions a dataset path or DataFrame variable and asks to "profile" it
• User mentions wanting to understand the shape, quality, or distributions

Do NOT activate this skill for:

• Pure modeling questions (use feature-engineer or model-trainer instead)
• Quick one-off questions about a single column (just answer directly)
• Datasets with > 1M rows without sampling first

Workflow

For every dataset, follow these steps in order:

1. Load & inspect — read the file with the right pandas reader, check shape and dtypes
2. Run all 9 sections of the standard report (below) — never skip a section
3. Output as a single markdown report with clear ## headers per section
4. End with concrete recommendations, not generic advice

Standard EDA Report Structure

The report must have exactly these 9 sections, in this order:

1. Dataset Overview

• File name, file size on disk, file format
• Shape: (n_rows, n_cols)
• Memory footprint (df.memory_usage(deep=True).sum() in MB)
• A small head(5) preview

2. Schema & Types

• Table with: column name, dtype, n_unique, n_null, sample_value
• Flag columns where n_unique == n_rows (likely IDs)
• Flag columns where n_unique == 1 (constant — useless)

3. Missing Values

• Total count and percentage missing per column
• Sort descending; only show columns with > 0 missing
• If any column > 50% missing → flag as "DROP candidate"
• Missingness pattern: are missings concentrated in specific rows?

4. Descriptive Statistics

• For numeric: df.describe() plus skewness and kurtosis
• For categorical: top 5 most frequent values per column with counts and %
• For datetime: min, max, range in days

5. Distributions

• For each numeric column, generate a histogram (use matplotlib, no seaborn)
• For each categorical column with cardinality < 20, generate a bar chart of top values
• Save plots to `outputs/eda/` as PNG, embed in the report

6. Correlations

• Compute df.select_dtypes(include='number').corr() (Pearson by default)
• Generate a heatmap (matplotlib imshow)
• List the top 10 absolute correlations (excluding diagonal) as a table
• Flag any correlation > 0.95 as "potentially redundant features"

7. Outliers

• For each numeric column, count outliers using both:
• IQR method: outside [Q1 - 1.5*IQR, Q3 + 1.5*IQR]
• Z-score method: |z| > 3
• Report counts and percentages
• Flag columns where outliers exceed 5% of rows

8. Data Quality Score

• Compute a score on 100 with these components:
• Completeness: 100 - mean(% missing) → max 30 points
• Uniqueness: penalize duplicate rows → max 20 points
• Validity: % of values within expected ranges (use IQR bounds as proxy) → max 25 points
• Consistency: penalize mixed types in object columns → max 25 points
• Output: Score: XX/100 [GREEN | YELLOW | RED]
• GREEN = 80+, YELLOW = 60-79, RED = below 60
• List the 3 main red flags even if the score is GREEN

9. Recommendations

• 3 to 7 specific, actionable recommendations — never generic
• Each recommendation must reference a specific column or pattern observed above
• Categorize as: [CLEAN], [IMPUTE], [DROP], [ENGINEER], [INVESTIGATE]

Code Template

Use this template as the backbone. Adapt to the actual data, but keep the section structure identical.

import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
from pathlib import Path
from scipy import stats

# Setup
OUTPUT_DIR = Path("outputs/eda")
OUTPUT_DIR.mkdir(parents=True, exist_ok=True)

# 1. Load
df = pd.read_csv(filepath)  # or read_excel, read_parquet
print(f"Loaded {df.shape[0]:,} rows × {df.shape[1]} cols")

# 2. Schema
schema = pd.DataFrame({
    "dtype": df.dtypes,
    "n_unique": df.nunique(),
    "n_null": df.isna().sum(),
    "sample": df.iloc[0],
})

# 3. Missing
missing = df.isna().sum().sort_values(ascending=False)
missing_pct = (missing / len(df) * 100).round(2)

# 4. Describe
numeric_describe = df.describe()
cat_cols = df.select_dtypes(include="object").columns
cat_describe = {col: df[col].value_counts().head(5) for col in cat_cols}

# 5. Distributions (one PNG per numeric col)
for col in df.select_dtypes(include="number").columns:
    fig, ax = plt.subplots(figsize=(6, 3))
    df[col].hist(bins=30, ax=ax)
    ax.set_title(f"Distribution: {col}")
    fig.tight_layout()
    fig.savefig(OUTPUT_DIR / f"hist_{col}.png", dpi=80)
    plt.close(fig)

# 6. Correlations
corr = df.select_dtypes(include="number").corr()

# 7. Outliers
def count_outliers(s):
    q1, q3 = s.quantile([0.25, 0.75])
    iqr = q3 - q1
    iqr_count = ((s < q1 - 1.5 * iqr) | (s > q3 + 1.5 * iqr)).sum()
    z_count = (np.abs(stats.zscore(s.dropna())) > 3).sum()
    return iqr_count, z_count

# 8. Quality score (see formula in section 8 above)
# 9. Recommendations (generated from observations)

Output format

The final response to the user must be a single markdown document with:

# EDA Report — <filename>

## 1. Dataset Overview
...
## 2. Schema & Types
...
## 3. Missing Values
...
## 4. Descriptive Statistics
...
## 5. Distributions
[![Histogram column_x](outputs/eda/hist_column_x.png)]
...
## 6. Correlations
...
## 7. Outliers
...
## 8. Data Quality Score
**Score: 78/100 [YELLOW]**
Top red flags: ...
## 9. Recommendations
1. [CLEAN] Column `email` has 12 mixed-case duplicates — normalize to lowercase
2. [IMPUTE] Column `age` has 8% missing — consider median imputation grouped by `region`
3. ...

Constraints

• Never use seaborn (matplotlib only — keeps dependencies light)
• Never modify the source dataframe
• If the dataset has > 100k rows, sample to 100k for plots but compute all stats on the full data
• Keep plot DPI at 80 (lightweight files for the report)
• Output language: match the language the user wrote in (FR or EN)