4. Feature stores in training pipelines

Training-serving skew is one of the most persistent failure modes in production ML. A model trained on features computed by a batch Spark job may see entirely different values at inference time, where the same feature is computed by a low-latency service reading from a key-value store. The feature store exists to eliminate this class of bug by defining each feature once and materializing it into two synchronized representations: an offline store for point-in-time-correct training datasets, and an online store for low-latency serving. Feast and Tecton are the two dominant implementations of this pattern, and they differ substantially in how much of the pipeline they own.

Why a Feature Store Belongs in the Training Pipeline

A training pipeline that reads raw events directly from a warehouse has three recurring problems. First, temporal leakage: joining a label at time t with a feature value computed at time t + 3h produces a model that looks excellent offline and fails in production. Second, duplicated logic: the batch SQL and the serving code drift apart because they are maintained by different teams. Third, no reuse: every new model re-derives the same aggregates from scratch.

A feature store addresses all three by introducing a declarative feature definition, a point-in-time join engine for offline retrieval, and a materialization path that pushes the same computed values into an online store. The training pipeline then consumes a training dataset produced by the store rather than hand-written joins.

Feast: Lightweight, Warehouse-Native, Bring-Your-Own-Compute

Feast is an open-source feature store that deliberately does not own your transformation logic. You define entities, data sources, and feature views in Python, and Feast handles the offline/online split, point-in-time joins, and materialization. The actual computation stays in whatever system you already use — dbt, Spark, Flink, or a scheduled Python job.

from feast import Entity, FeatureView, Field, FileSource
from feast.types import Float32, Int64
from datetime import timedelta

driver = Entity(name="driver_id", join_keys=["driver_id"])

driver_stats_source = FileSource(
    name="driver_stats_source",
    path="s3://ml-features/driver_stats.parquet",
    timestamp_field="event_timestamp",
)

driver_stats_fv = FeatureView(
    name="driver_hourly_stats",
    entities=[driver],
    ttl=timedelta(hours=24),
    schema=[
        Field(name="conv_rate", dtype=Float32),
        Field(name="avg_daily_trips", dtype=Int64),
    ],
    source=driver_stats_source,
)

The critical operational step is feast materialize, which reads the most recent value per entity key from the offline source and writes it into the online store (Redis, DynamoDB, or SQLite for local development):

feast materialize-incremental $(date -u +"%Y-%m-%dT%H:%M:%S")

For training, get_historical_features performs the point-in-time join. Given an entity dataframe with driver_id and event_timestamp columns, Feast returns, for each row, the feature values as they existed at that timestamp — respecting the ttl so that stale rows are excluded rather than silently reused.

Tecton: Managed Transformations and Streaming Materialization

Tecton takes the opposite architectural stance: it owns the transformation layer. Features are defined as either batch, stream, or on-demand transformations, and Tecton compiles each into the appropriate execution engine. A single logical feature can be backed by a Spark job for backfill and a Flink job for real-time updates, with the platform guaranteeing that both produce identical values.

@batch_feature_view(
    sources=[driver_events],
    entities=[driver],
    mode="spark_sql",
    timestamp_field="event_timestamp",
    online=True,
    offline=True,
    feature_start_time=datetime(2024, 1, 1),
)
def driver_hourly_stats(driver_events):
    return f"""
        SELECT driver_id,
               event_timestamp,
               SUM(trips) / 3600.0 AS conv_rate,
               COUNT(*) AS avg_daily_trips
        FROM {driver_events}
        GROUP BY driver_id, event_timestamp
    """

Because Tecton controls both the batch and streaming paths, it can guarantee consistency without relying on the user to keep two codebases in sync. The trade-off is operational coupling: you adopt Tecton's SDK, its materialization schedule, and its runtime.

Choosing Between Them

Wiring the Store into the Training Pipeline

The training pipeline's job is to produce a reproducible dataset and a model artifact whose feature dependencies are explicit. A practical sequence looks like this:

1. Build an entity dataframe from the label source: one row per (entity key, label timestamp). This is the spine of the training set.
2. Call get_historical_features (Feast) or get_features_for_events (Tecton) to attach feature values at those timestamps.
3. Persist the resulting dataset with a content hash and the feature view versions used, so the exact training set can be reconstructed.
4. Train, evaluate, and register the model alongside its feature references.
5. Before deployment, verify that the online store returns values consistent with the offline store for a sample of entity keys.

Step 5 is the one teams most often skip, and it is where skew actually surfaces. A consistency check compares the offline value at the latest timestamp against the online value for the same key:

offline = store.get_historical_features(
    entity_df=pd.DataFrame({"driver_id": [101, 102], "event_timestamp": [now, now]}),
    features=["driver_hourly_stats:conv_rate"],
).to_df()

online = store.get_online_features(
    features=["driver_hourly_stats:conv_rate"],
    entity_rows=[{"driver_id": 101}, {"driver_id": 102}],
).to_dict()

# Compare; a mismatch beyond float tolerance indicates materialization lag or logic drift.

What the Feature Store Does Not Solve

A feature store guarantees that the same definition produces the same value in both paths. It does not guarantee that the definition is correct, that upstream data is fresh, or that the label is aligned with the feature window. Those remain pipeline responsibilities. The store's value is narrow but decisive: it removes an entire category of silent, hard-to-diagnose skew from the training-to-serving boundary.

The diagram above shows the essential contract: one feature definition feeds two materialization paths, and the consistency check is what closes the loop between them. Whether you implement that contract with Feast's bring-your-own-compute model or Tecton's managed transformations depends on how much of the transformation layer you want the platform to own — but the contract itself is non-negotiable if you want training and serving to agree.