3. Detect Data Leakage Between Simulation Batches in a Surrogate Dataset

Why batch-level leakage hides in surrogate datasets

Simulation campaigns rarely produce independent samples. A single DOE run often generates a batch of load cases that share geometry, mesh topology, material card, solver version, and sometimes only differ in a scalar load multiplier. When those batches are split randomly into training and test partitions, near-duplicate runs land on both sides of the split. The surrogate then interpolates between points it has effectively already seen, and validation scores look far better than the model's true generalization ability. This case builds a concrete detector for that condition and quantifies the optimistic bias it introduces.

The task has two coupled parts: (1) find near-duplicate simulation runs that straddle the train/test boundary using feature-space similarity, and (2) measure how much the reported validation metric improves because of those duplicates. We work with a tabular surrogate dataset whose rows are simulation runs and whose columns are normalized input parameters plus a scalar output (e.g., peak tip deflection).

Constructing a reproducible leakage scenario

To make the diagnostic verifiable, we synthesize a dataset with a known leakage structure. Each "batch" is a base parameter vector; runs inside a batch perturb only a small subset of features by a tiny amount. A random split will scatter batch members across partitions, which is exactly the failure mode we want to detect.

import numpy as np
import pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.ensemble import GradientBoostingRegressor
from sklearn.metrics import r2_score
from sklearn.neighbors import NearestNeighbors

rng = np.random.default_rng(7)

# 40 batches x 6 runs each = 240 simulation runs
n_batches, runs_per_batch = 40, 6
feature_names = ["E", "rho", "L", "t", "load", "mesh_density"]

base = rng.uniform(low=[190e9, 7800, 0.8, 0.004, 500, 2.0],
                   high=[210e9, 7900, 1.2, 0.008, 5000, 6.0],
                   size=(n_batches, len(feature_names)))

rows, batch_ids = [], []
for b in range(n_batches):
    for r in range(runs_per_batch):
        # perturb only load and mesh_density: near-duplicate within batch
        jitter = np.zeros(len(feature_names))
        jitter[4] = rng.normal(0, 0.01 * base[b, 4])
        jitter[5] = rng.normal(0, 0.02 * base[b, 5])
        rows.append(base[b] + jitter)
        batch_ids.append(b)

X = np.array(rows)
# Synthetic output: smooth physics-like response + small noise
y = (X[:, 4] * X[:, 2]**3 / (48 * X[:, 0] * X[:, 3]**3 * 1e-6)
     + 0.05 * X[:, 5] + rng.normal(0, 0.02, len(X)))

df = pd.DataFrame(X, columns=feature_names)
df["batch_id"] = batch_ids
df["deflection"] = y
print(df.shape)  # (240, 8)

Each batch shares five of six features almost exactly; only load and mesh_density vary slightly. A naive random split will place sibling runs in both partitions with high probability, which is the leakage we want to expose.

Detecting near-duplicates across the split

We standardize features, fit a nearest-neighbor index on the training partition, and query each test row for its closest training neighbor. A test run whose nearest training neighbor sits within a small relative distance is a leakage candidate. The threshold must be chosen in feature space, not output space, so the detector does not depend on the surrogate being correct.

from sklearn.preprocessing import StandardScaler

Xtr, Xte, ytr, yte, btr, bte = train_test_split(
    df[feature_names].values, df["deflection"].values,
    df["batch_id"].values, test_size=0.3, random_state=0)

scaler = StandardScaler().fit(Xtr)
Xtr_s, Xte_s = scaler.transform(Xtr), scaler.transform(Xte)

nn = NearestNeighbors(n_neighbors=1, metric="euclidean").fit(Xtr_s)
dist, idx = nn.kneighbors(Xte_s)
dist = dist.ravel(); idx = idx.ravel()

# Relative distance in standardized space; 0.05 ~ 5% of a unit std
threshold = 0.05
leak_mask = dist < threshold
print("test rows:", len(Xte), "flagged:", leak_mask.sum())

Two independent signals confirm the flag is meaningful: the flagged test rows should overwhelmingly share a batch_id with their matched training neighbor, and their nearest-neighbor distance should be an order of magnitude smaller than the median. Both checks are cheap and catch threshold mistakes.

same_batch = bte[leak_mask] == btr[idx[leak_mask]]
print("flagged rows sharing batch:", same_batch.mean())   # expect ~1.0
print("median NN dist (all):", np.median(dist))
print("median NN dist (flagged):", np.median(dist[leak_mask]))

Quantifying the optimistic bias

Now train the same surrogate twice: once on the full training partition (leaky), and once after removing training rows whose batch also appears in the test partition (clean). Compare test R² on the identical test set. The gap is the optimistic bias attributable to batch-level leakage.

def fit_eval(Xtr, ytr, Xte, yte):
    m = GradientBoostingRegressor(n_estimators=300, max_depth=3,
                                  learning_rate=0.05, random_state=0)
    m.fit(Xtr, ytr)
    return r2_score(yte, m.predict(Xte))

r2_leaky = fit_eval(Xtr, ytr, Xte, yte)

# Clean split: drop training rows whose batch appears in test
leaky_batches = set(bte)
keep = ~np.isin(btr, list(leaky_batches))
r2_clean = fit_eval(Xtr[keep], ytr[keep], Xte, yte)

print(f"R2 leaky = {r2_leaky:.4f}")
print(f"R2 clean = {r2_clean:.4f}")
print(f"optimistic bias = {r2_leaky - r2_clean:+.4f}")

On this synthetic dataset the leaky split typically reports R² near 0.99 while the clean split drops to roughly 0.85–0.92, a bias of several hundredths of R². The exact numbers depend on the random seed and the noise level; treat them as expected magnitudes, not fixed results. The important point is the direction and the mechanism: the leaky model is rewarded for memorizing batch-local perturbations, not for generalizing across the parameter space.

Reading the diagnostic

The SVG below summarizes the decision logic and the two outputs the detector must produce: a leakage flag per test row and a bias estimate from the paired evaluation.

Interpreting and acting on the result

Three outcomes are worth distinguishing. If the flagged fraction is near zero and the bias is small, the split is effectively batch-clean and the reported validation score can be trusted. If the flagged fraction is high and the bias is large, the split is contaminated and the leaky score must be discarded. If the flagged fraction is high but the bias is small, the surrogate is not exploiting the duplicates — still worth fixing the split, but the reported metric is not badly inflated.

The detector is deliberately independent of the surrogate: it operates only on input features and batch metadata, so it can be run before any model is trained. That ordering matters — if you train first and then look for leakage, you risk rationalizing a suspiciously good score. Run the detector, fix the split, then train and report. The bias estimate from the paired evaluation is the number to carry forward into later error diagnostics, because it tells you how much of the earlier baseline was an artifact of the split rather than a property of the surrogate.