1. Simulation data leakage fundamentals
In simulation-based development, a model is only as trustworthy as the data it was trained and validated on. Simulation data leakage occurs when information that would not be legitimately available at inference time—or information that overlaps between training and evaluation sets—contaminates the learning or testing process. The result is a model that appears to generalize well during development but degrades sharply when deployed. This chapter establishes a precise definition, a working taxonomy, and the mechanisms by which leakage corrupts generalization testing, so that later chapters can build detection and prevention procedures on a shared vocabulary.
Defining Leakage in a Simulation Context
Data leakage is any situation in which the training pipeline has access to information that is unavailable, or would be unavailable, at prediction time, or in which evaluation data shares hidden dependencies with training data. In simulation workflows the problem is amplified because synthetic datasets are generated in bulk: a single scenario generator can emit thousands of near-duplicate trajectories, and a naive random split will scatter those near-duplicates across train, validation, and test partitions.
A useful formal framing: let D be the full simulated dataset, f the model, and S a splitting function producing Dtrain and Dtest. Leakage exists when the mutual information between Dtrain and Dtest is non-negligible through channels other than the intended task signal—shared random seeds, shared scenario parameters, shared sensor noise realizations, or shared preprocessing statistics. The model then exploits those channels, and the test metric measures memorization of the generator rather than generalization to the task.
A Taxonomy of Leakage Sources
Leakage in simulation pipelines rarely has a single cause. The taxonomy below separates the mechanisms so that each can be audited independently. The accompanying diagram maps these categories onto a typical simulation-to-model pipeline, showing where each leak enters and which evaluation artifact it corrupts.
Generator leakage. The scenario generator itself becomes a shared information channel. If the same random seed, the same road-network instance, or the same sensor-noise realization is reused across partitions, samples that look different at the surface are statistically coupled. A model can learn generator-specific fingerprints—for example, a particular pattern of LiDAR dropout—and score well on test data drawn from the same generator run.
Split leakage. Random splitting of temporally or spatially correlated simulations places near-duplicate frames on both sides of the boundary. Consider a 30-second driving log sampled at 10 Hz: consecutive frames differ by centimeters. A random 80/20 split will put frame t in training and frame t+1 in test, so the test set is effectively a copy of the training set. Group-aware splitting by scenario, by map tile, or by time window is the standard remedy.
Preprocessing leakage. Feature scaling, imputation, and target encoding computed on the full dataset before splitting leak distributional statistics from test into train. A concrete symptom: normalization using the global mean and standard deviation of all simulated runs makes the test set's statistics visible during training. The fix is to fit every transformer on the training partition only and apply the frozen parameters to validation and test.
Target and temporal leakage. Features that encode the label or future state—such as a "collision flag" derived from the same simulation step used to define the target, or a post-hoc annotation channel—give the model an illegitimate shortcut. In time-series simulation, using future frames to predict past events is the temporal variant.
Why Leakage Corrupts Generalization Testing
Generalization testing asks a specific question: how does the model perform on data drawn from a distribution it has never seen? Leakage answers a different question—how well does the model memorize the generator? The gap between these two questions is the leakage-induced optimism bias.
Consider a concrete example. A perception model is trained on 50,000 simulated frames from 200 scenarios and evaluated with a random frame-level split. Reported validation accuracy is 94.2%. When the same model is re-evaluated with a scenario-level split—no scenario appearing in both partitions—accuracy drops to 78.5%. The 15.7-point gap is not a modeling failure; it is the measurable cost of split leakage. The first number was never a generalization estimate.
Leakage also distorts model selection. When validation and test share leaked information, hyperparameter search and early stopping optimize for the leaked channel. The selected model may be the one that best exploits the leak rather than the one that generalizes best, so the corruption compounds across the development cycle.
Practical Audit: Reproducible Leakage Check
The following procedure is a minimal, reproducible audit that can be run before any generalization claim is made. It assumes a dataset of simulated episodes, each with a scenario identifier and a time index.
# Inputs (example parameters, illustrative only)
# episodes.parquet : columns [episode_id, scenario_id, t, features..., label]
# split_ratio : 0.8 / 0.1 / 0.1
1. Group-aware split
- Partition by scenario_id, not by row.
- Assert: set(train.scenario_id) ∩ set(test.scenario_id) == ∅
2. Near-duplicate scan
- For each test episode, compute max cosine similarity
against all train episodes on the feature vector.
- Flag any pair with similarity > 0.98 as a suspected duplicate.
3. Preprocessing isolation
- Fit scaler on train only; transform val/test with frozen params.
- Verify: train_mean != global_mean when distributions differ.
4. Temporal boundary check
- For time-series episodes, confirm no test window overlaps
a train window in the same scenario.
5. Report
- random_split_score, group_split_score, gap = random - group
- duplicate_count, overlap_count
Expected observable outcomes: a clean dataset yields a small gap (typically under 2–3 points for stable tasks) and zero flagged duplicates. A leaky dataset shows a large gap, non-zero duplicate counts, or overlapping scenario identifiers. Typical failure symptoms and their interpretations:
- Gap > 10 points with zero duplicates: likely preprocessing leakage or a shared generator seed; inspect scaler fitting order and seed management.
- Duplicate count > 0: split leakage; switch to scenario- or map-tile-level grouping.
- Overlap count > 0: temporal leakage; enforce time-window separation within each scenario.
- Gap small but absolute score implausibly high: target leakage; audit feature provenance for label-derived channels.
Impact Summary and Design Implications
The practical consequence is that every generalization claim must be paired with an explicit statement of the splitting unit and the preprocessing boundary. A model evaluated on a random frame split has demonstrated interpolation within seen scenarios; a model evaluated on held-out scenarios has demonstrated something closer to generalization. These are different claims and must not be reported interchangeably.
Three design rules follow directly from the taxonomy. First, define the splitting unit before generating data, and make it the coarsest unit that still leaves enough samples—scenario, map tile, or time block. Second, treat all preprocessing as part of the model and fit it inside the training fold. Third, maintain a provenance record for every feature so that label-derived or future-derived channels can be identified and removed. These rules are inexpensive to adopt at the start of a project and expensive to retrofit after a leakage-inflated result has already shaped the design.
With the definition, taxonomy, and impact established, subsequent chapters can move from diagnosis to prevention: constructing leakage-resistant splits, instrumenting pipelines to detect contamination automatically, and designing generalization tests whose reported numbers survive scrutiny on genuinely unseen simulation conditions.