ML Debugging

Overview

ML bugs are usually silent — the code runs, but the model is wrong. Debug systematically by isolating which layer is broken: data, optimization, or generalization. Form a hypothesis, test it with the cheapest possible experiment, then move on.

When to use

• Loss is NaN/Inf or stuck.
• Metrics are implausibly high (leakage) or stuck at chance.
• Train/val curves diverge.

First move: can it overfit one batch?

The fastest sanity check in ML. Take a single small batch and train until loss ≈ 0.

x, y = next(iter(train_loader))
for _ in range(200):
    optimizer.zero_grad()
    loss = criterion(model(x), y)
    loss.backward(); optimizer.step()
print(loss.item())  # should approach 0

• Can't overfit one batch → bug in model/loss/labels/data wiring (not capacity).
• Overfits one batch but not the dataset → optimization or regularization issue.

Symptom → cause decision tree

Leakage hunt (when results are "too good")

1. Any feature with |corr| ≈ 1.0 to target?
2. Is preprocessing fit before the split?
3. Do rows from one entity span train and test? (use GroupKFold)
4. Time series shuffled? (must be chronological)
5. Is a post-outcome column (e.g., refund_issued) used to predict the outcome?

Optimization checklist

• Normalize/standardize inputs.
• Sweep LR over [1e-5 ... 1e-1] (log scale) — wrong LR causes most failures.
• Add gradient clipping for RNNs/transformers.
• Check for NaNs in inputs/targets before training.
• Verify loss matches task (CE for classification logits, not MSE).

Pitfalls

• Changing five things at once — change one variable per experiment.
• Debugging on the full dataset — shrink to fail fast.
• Trusting metrics without inspecting predictions/confusion matrix.

Hand-off

A root-caused fix plus a regression check (the one-batch test + a held-out metric) so experiment-tracking can confirm the issue stays fixed.