ML Feature Engineering Patterns

The Data Engineering / ML Boundary

Data engineers own the data. ML engineers own the models. Feature engineering sits at the boundary — and when it's designed poorly, both sides pay for it. The most common failures:

1. Training/serving skew — features computed differently at training time vs. prediction time → model performs worse in production than in validation
2. Label leakage — features computed using data from the future, making the training set artificially easy → model fails in production
3. Feature duplication — each ML team recomputes the same features independently → inconsistent definitions, wasted compute

The patterns in this skill address all three.

Point-in-Time Correct Joins

This is the most important concept in ML feature engineering. A model trained to predict churn should only "see" data that would have been available at the time of the prediction — not data from the future.

The wrong way

-- BAD: This leaks future data into training
-- If we're predicting churn as of 2024-01-15, we shouldn't know about
-- events that happened on 2024-01-20
SELECT
    u.user_id,
    u.subscription_tier,
    COUNT(e.event_id) AS events_last_30d,  -- counts events AFTER the label date!
    u.churned AS label
FROM users u
JOIN events e ON u.user_id = e.user_id
    AND e.event_date >= DATEADD(day, -30, CURRENT_DATE)  -- wrong: uses today's date
WHERE u.label_date = '2024-01-15'
GROUP BY 1, 2, 4;

The right way — as-of join

-- GOOD: Features are computed as of the label date, using only past data
SELECT
    u.user_id,
    u.subscription_tier,
    COUNT(e.event_id) AS events_last_30d,
    u.churned AS label
FROM user_labels u  -- contains (user_id, label_date, churned)
LEFT JOIN events e ON u.user_id = e.user_id
    AND e.event_date >= DATEADD(day, -30, u.label_date)  -- relative to label date
    AND e.event_date < u.label_date                       -- only past events
GROUP BY 1, 2, 4;

The rule: every date range in a feature join must be bounded by < label_date on the upper end.

Point-in-time with slowly-changing dimensions

When a dimension table changes over time (e.g., a user's subscription tier changes), you need the value that was true at label_date, not the current value.

-- Get the subscription tier each user had at each label date
-- (assuming subscription_changes has valid_from/valid_to columns)
SELECT
    l.user_id,
    l.label_date,
    s.subscription_tier AS tier_at_label_date
FROM user_labels l
JOIN subscription_changes s ON l.user_id = s.user_id
    AND s.valid_from <= l.label_date
    AND (s.valid_to > l.label_date OR s.valid_to IS NULL)

Feature Pipeline Architecture

A feature pipeline has two paths:

• Offline — batch computation, used for training and bulk scoring
• Online — low-latency lookup, used for real-time predictions

Offline pipeline:
  Raw events → Transform → Feature store (offline) → Training dataset → Model training

Online pipeline:
  Raw events → Transform → Feature store (online) → Real-time prediction API

The transform logic must be identical in both paths. If it isn't, you get training/serving skew.

The serving skew trap

# Training time (Python/Pandas)
df["days_since_login"] = (df["label_date"] - df["last_login_date"]).dt.days

# Serving time (different code, different team)
days_since_login = (datetime.now() - user.last_login).days  # subtly different

Any divergence in feature computation between training and serving will degrade model performance in production. The model won't fail — it will just quietly perform worse.

Prevention: Keep feature logic in a single, shared function called by both the training pipeline and the serving code. Write integration tests that verify identical output for both paths on the same input.

# Shared feature computation — called by BOTH training pipeline and serving
def compute_engagement_features(user_id: str, as_of_date: date) -> dict:
    events = db.query("""
        SELECT COUNT(*) AS event_count,
               MAX(event_date) AS last_event_date
        FROM events
        WHERE user_id = %s AND event_date < %s
          AND event_date >= %s
    """, user_id, as_of_date, as_of_date - timedelta(days=30))

    return {
        "events_last_30d": events["event_count"],
        "days_since_last_event": (as_of_date - events["last_event_date"]).days
            if events["last_event_date"] else None
    }

Feature Store Patterns

A feature store is a system that manages feature computation, storage, and serving. Whether you use a managed one (Feast, Tecton, Hopsworks) or a simple internal one, the key components are the same.

When you need a feature store

• Multiple models need the same features (churn model, LTV model, propensity model all need engagement features)
• Features need to be available both for offline training and online serving
• You want a catalog of what features exist and who owns them
• You need to backfill features for historical label dates

If only one model exists and it only runs in batch, you may not need a full feature store — a well-structured dbt model is sufficient.

Simple internal feature store (dbt + BigQuery)

-- The feature table — one row per (entity, feature_date)
-- Materialized as an incremental dbt model
-- models/feature_store/user_engagement_features.sql

{{
  config(
    materialized='incremental',
    unique_key=['user_id', 'feature_date'],
    partition_by={'field': 'feature_date', 'data_type': 'date'},
  )
}}

SELECT
    user_id,
    DATE(event_date) AS feature_date,
    COUNT(*) AS events_last_30d,
    COUNT(DISTINCT session_id) AS sessions_last_30d,
    MAX(event_date) AS last_event_date,
    DATE_DIFF(DATE(event_date), MAX(event_date), DAY) AS days_since_last_event
FROM {{ ref('stg_events') }}
{% if is_incremental() %}
WHERE event_date >= DATE_SUB((SELECT MAX(feature_date) FROM {{ this }}), INTERVAL 31 DAY)
{% endif %}
GROUP BY 1, 2

Training time: query the feature table with a point-in-time join against your label table. Serving time: query the feature table for feature_date = today().

Feast (managed feature store)

from feast import FeatureStore, Entity, Feature, FeatureView, FileSource
from feast.types import Float64, Int64

# Define entity
user = Entity(name="user_id", join_keys=["user_id"])

# Define feature view
user_engagement_fv = FeatureView(
    name="user_engagement",
    entities=[user],
    ttl=timedelta(days=1),
    features=[
        Feature(name="events_last_30d", dtype=Int64),
        Feature(name="days_since_last_event", dtype=Int64),
    ],
    source=BigQuerySource(table="project.dataset.user_engagement_features"),
)

store = FeatureStore(repo_path=".")

# Training — get historical features (point-in-time correct automatically)
training_df = store.get_historical_features(
    entity_df=label_df,  # contains user_id and event_timestamp (the label date)
    features=["user_engagement:events_last_30d", "user_engagement:days_since_last_event"],
).to_df()

# Serving — get online features (millisecond latency from Redis/DynamoDB)
features = store.get_online_features(
    features=["user_engagement:events_last_30d"],
    entity_rows=[{"user_id": "user_123"}],
).to_dict()

Label Design

Churn label example

-- Define churn: no activity for 30 days after a 90-day observation window
-- Label date = the date we're making the prediction
-- We look 30 days forward from label_date to determine if they churned

SELECT
    user_id,
    label_date,
    CASE
        WHEN MAX(event_date) < DATEADD(day, 30, label_date) THEN 1
        ELSE 0
    END AS churned
FROM (
    -- Generate label dates (one per user per week)
    SELECT DISTINCT user_id, date_spine.date AS label_date
    FROM active_users
    CROSS JOIN date_spine
    WHERE date_spine.date BETWEEN '2023-01-01' AND '2023-12-31'
) label_candidates
LEFT JOIN events e ON label_candidates.user_id = e.user_id
    AND e.event_date BETWEEN label_date AND DATEADD(day, 30, label_date)
GROUP BY 1, 2

Important: label generation requires future data. The label for a given label_date can only be computed 30 days later. Don't use recent dates as labels in training — the churned flag won't be set yet.

Feature Documentation for ML Teams

When data engineers hand features to ML teams, document the grain, computation window, and known caveats:

# feature_catalog/user_engagement.yaml
feature_group: user_engagement
entity: user_id
grain: "One row per (user_id, feature_date)"
update_frequency: daily
owner: [email protected]

features:
  - name: events_last_30d
    type: integer
    description: "Count of all events in the 30 days prior to feature_date"
    computation_window: "feature_date - 30 days to feature_date (exclusive)"
    known_caveats: "Null for users who signed up < 30 days before feature_date"

  - name: days_since_last_event
    type: integer
    description: "Days between feature_date and the most recent event before feature_date"
    computation_window: "All history up to feature_date"
    known_caveats: "Null for users with no recorded events"

Checklist: Before Handing Features to the ML Team

• [ ] Point-in-time correctness verified — all feature windows are bounded by < label_date
• [ ] Feature computation function is shared between training pipeline and serving code
• [ ] Null values documented — what does a null mean for each feature?
• [ ] Feature distribution checked after backfill (spot-check means and percentiles per month)
• [ ] Label leakage audit — no feature references data that would only be known after the label event
• [ ] Features versioned — if computation logic changes, old features remain accessible
• [ ] Feature documentation written (grain, window, caveats, owner)
• [ ] Integration test confirms offline and online paths produce identical output for same input