2. Leakage sources in simulation pipelines

Simulation data leakage is not a single defect that appears at one point in a pipeline; it is a family of failure modes that enter at different stages and propagate downstream. A model trained on a simulation dataset can appear to generalize well during validation while collapsing on deployment, and the root cause is often not the learning algorithm but the way scenarios were generated, parameters were sampled, or features were preprocessed. This chapter isolates the three dominant leakage surfaces in a simulation pipeline — scenario generation, parameter sampling, and data preprocessing — and shows how each one can silently inject information that would not be available at inference time.

The unifying principle is straightforward: any information that flows from the evaluation target, the future, or the held-out distribution into the training pipeline constitutes leakage. In simulation settings this principle is harder to enforce than in static datasets because the pipeline itself is generative. Scenarios are synthesized, parameters are drawn from distributions, and features are computed from raw signals. Each of these stages is a place where the boundary between "training-available" and "evaluation-only" information can be crossed, often without any explicit error.

Scenario generation as a leakage surface

Scenario generation defines the operating conditions under which simulation episodes are produced: road topology, traffic density, weather, sensor noise profiles, adversarial agent behavior, or fault injection schedules. Leakage enters when the scenario set used for training is not independent of the scenario set used for evaluation, or when scenario identifiers themselves encode the label.

A common and subtle case is scenario-family overlap. Suppose a pipeline generates 500 episodes from 40 underlying scenario templates, then splits episodes randomly into train and test. Because multiple episodes share the same template, the test set contains near-duplicates of training episodes. The model memorizes template-specific dynamics rather than learning transferable behavior. The correct split is at the template level, not the episode level: all episodes derived from a given template must land in exactly one partition. The same logic applies to map tiles, weather seeds, and agent behavior scripts — the split unit must be the highest-level entity that is shared across episodes.

A second pattern is label-conditioned generation. If the scenario generator is configured to produce "hard braking events" only for the positive class and "normal cruising" only for the negative class, then the scenario metadata (duration, speed profile, agent count) becomes a proxy for the label. A classifier can achieve high accuracy by reading these proxies instead of the intended sensor features. The diagnostic is to train a model on metadata alone; if metadata-only accuracy is far above the base rate, the generator is leaking.

Parameter sampling and distributional leakage

Parameter sampling draws the continuous and discrete variables that configure each simulation: mass, friction coefficient, sensor bias, latency, wind speed, payload, or controller gains. Leakage arises when the sampling distribution used for training is informed by the evaluation set, or when the sampling procedure itself is not reproducible across partitions.

Consider a pipeline that tunes its parameter ranges by running a preliminary sweep and keeping only the ranges that produce "interesting" outcomes. If that sweep is performed on the full dataset before splitting, the resulting ranges are conditioned on all episodes, including those later assigned to test. The test set is no longer a clean holdout. The fix is to perform any range selection, outlier filtering, or difficulty calibration strictly inside the training partition, using a nested split.

Reproducibility is a related but distinct issue. If parameter sampling uses a global random seed that is advanced across the entire pipeline, then the order in which episodes are generated determines their parameters. Shuffling or reordering episodes after generation can therefore correlate parameters with split assignment. The practical remedy is to derive a per-episode seed from a stable identifier (for example, a hash of the scenario template ID and episode index) so that parameter draws are independent of processing order.

# Deterministic per-episode parameter sampling
import hashlib, numpy as np

def episode_seed(template_id: str, episode_idx: int, salt: str = "v1") -> int:
    key = f"{template_id}:{episode_idx}:{salt}".encode()
    return int.from_bytes(hashlib.sha256(key).digest()[:8], "big")

def sample_params(template_id, episode_idx, ranges):
    rng = np.random.default_rng(episode_seed(template_id, episode_idx))
    return {k: rng.uniform(lo, hi) for k, (lo, hi) in ranges.items()}

# Example: friction and sensor bias drawn independently of pipeline order
ranges = {"friction": (0.4, 0.9), "sensor_bias": (-0.02, 0.02)}
params = sample_params("urban_intersection_A", 17, ranges)
print(params)  # e.g. {'friction': 0.63, 'sensor_bias': 0.004}

With this scheme, regenerating the dataset in a different order yields identical parameters for each episode, and the train/test split cannot influence the draws. The salt string allows controlled regeneration when the parameterization changes, without silently reusing old draws.

Preprocessing leakage

Preprocessing is where leakage is most easily introduced and least easily noticed, because the operations look innocuous. Normalization statistics, imputation values, feature selection, dimensionality reduction, and resampling all estimate quantities from data. If those quantities are estimated on the full dataset before splitting, the training features contain information about the test distribution.

The canonical example is standardization. Computing the mean and standard deviation over all episodes and then applying them to both partitions leaks the test set's location and scale into training. The effect is small when distributions are stationary, but in simulation pipelines the test partition is often deliberately shifted (different weather, different sensor noise), so global statistics can encode the shift and inflate validation performance. The correct procedure fits the scaler on the training partition only and applies the frozen transform to validation and test.

Time-series simulation adds a temporal dimension. Windowed features that use future samples — a rolling mean centered on the current timestep, a filter with zero-phase response, or an interpolation that fills gaps using later observations — leak future information into each training window. The rule is causal: at timestep t, only samples up to t may contribute. Centered windows and zero-phase filters must be replaced with trailing windows and causal filters, or the evaluation must explicitly account for the look-ahead.

Resampling is another frequent offender. Oversampling minority failure events before splitting duplicates samples across partitions, so the test set contains exact copies of training rows. Undersampling can remove the very episodes that define the test distribution. Both operations must be applied after the split, and only to the training partition.

How the three surfaces interact

These leakage sources are not independent. A scenario generator that produces label-conditioned episodes will also skew the parameter distribution, and a globally fitted scaler will then encode that skew into every feature. The figure below traces how information flows from each stage into the training set and where the boundary must be enforced.

The diagram makes the enforcement point explicit: the split boundary must sit before any operation that estimates a quantity from data. Scenario and parameter stages are upstream of the boundary, so their outputs must be partitioned by template and seeded deterministically; preprocessing sits downstream, so every fitted transform must be trained on the training partition and frozen for evaluation.

A practical audit sequence

When validating a simulation pipeline, run the following checks in order. First, confirm the split unit: list the entities shared across episodes and verify that no entity appears in more than one partition. Second, run a metadata-only classifier on scenario and parameter fields; accuracy near the base rate indicates no proxy leakage, while high accuracy indicates label-conditioned generation or range tuning. Third, serialize the preprocessing object and refit it on the test partition; differing parameters confirm the transform was fitted on training data only. Fourth, inspect windowed features for any dependence on samples after the current timestep. Fifth, regenerate the dataset with shuffled episode order and confirm that per-episode parameters and split assignments are unchanged.

Each check targets a specific leakage surface and produces an observable signal rather than a subjective judgment. Together they convert leakage prevention from a vague aspiration into a repeatable gate that can be run before any generalization claim is made.