Snorkel Weak-Supervision Labeling

Overview

Hand-labeling clinical text is slow, expensive, and requires scarce expert time. Weak supervision flips the problem: experts encode their knowledge as labeling functions (LFs) noisy, partial heuristics and Snorkel's LabelModel learns each LF's accuracy and correlation to combine them into probabilistic training labels. You then train a normal classifier on those labels. This is the "training-set generation" approach for scaling supervised ML without a labeling army.

When to Use This Skill

• You need a labeled training set for medical text classification and have little/no gold.
• Subject-matter rules exist (keywords, ontologies, regexes, existing models) but no labels.
• You want to iterate on labels at the speed of code, with versioning and analysis.
• Bootstrapping labels for healthcare-predictive-modeling or a clinical NER/classifier.

Workflow

1. Define the task and label schema (e.g., note mentions active smoking: YES / NO /

ABSTAIN).

1. Write labeling functions keyword/regex, heuristic, ontology lookups (via

medical-ontology-code-mapping), assertion-aware rules (via clinical-nlp-entity- extraction), or weak model votes. Each LF returns a class or ABSTAIN.

1. Analyze LFs coverage, overlap, conflict, and empirical accuracy on a small dev

set; drop or fix low-value LFs.

1. Fit the LabelModel learn LF accuracies/correlations and output **probabilistic

labels** (no majority-vote naivety).

1. Train the end model train a classifier (transformer or TF-IDF + linear) on the

probabilistic labels, which generalizes beyond the LFs' coverage.

1. Evaluate the end model on a held-out gold set and iterate on weak LFs.

Example

from snorkel.labeling import labeling_function, PandasLFApplier, LFAnalysis
from snorkel.labeling.model import LabelModel
import re

ABSTAIN, NO, YES = -1, 0, 1

@labeling_function()
def lf_current_smoker(x):
    return YES if re.search(r"\b(current smoker|smokes daily|tobacco use)\b", x.text, re.I) else ABSTAIN

@labeling_function()
def lf_denies(x):
    return NO if re.search(r"\b(denies (tobacco|smoking)|non-?smoker|never smoker)\b", x.text, re.I) else ABSTAIN

@labeling_function()
def lf_former(x):
    return NO if re.search(r"\b(former smoker|quit smoking|ex-smoker)\b", x.text, re.I) else ABSTAIN

lfs = [lf_current_smoker, lf_denies, lf_former]
L = PandasLFApplier(lfs).apply(df)
print(LFAnalysis(L, lfs).lf_summary())        # coverage / overlaps / conflicts

label_model = LabelModel(cardinality=2)
label_model.fit(L, n_epochs=500, seed=42)
df["prob_label"] = label_model.predict_proba(L)[:, 1]   # train downstream model on these

Evaluation

Track LF coverage/conflict/accuracy during development and judge success only on the end model's metrics against a gold test set. Watch for correlated LFs that fool the LabelModel into overconfidence, and for systematic gaps where all LFs abstain (add LFs or collect a little gold there). Weak labels are noisy by design the win is scale.

Outputs

• probabilistic_labels.parquet example id, soft label, abstain mask.
• lf_analysis.csv per-LF coverage, overlap, conflict, empirical accuracy.
• weak_training_set.parquet ready for downstream model training.
• weak_supervision_report.md LF inventory, LabelModel quality, end-model metrics.

Healthcare Context

A strong fit for clinical text where labels are scarce but expert heuristics and ontologies are plentiful. Composes directly with clinical-nlp-entity-extraction (assertion-aware LFs) and medical-ontology-code-mapping (ontology LFs). Keep a real gold set for honest evaluation never report weak-label agreement as accuracy.

References

• Snorkel https://snorkel.org and https://github.com/snorkel-team/snorkel
• Ratner et al. (2017), Snorkel: Rapid Training Data Creation with Weak Supervision, VLDB.
• Ratner et al. (2016), Data Programming, NeurIPS.