1. Audit a Simulation Dataset for Duplicate Records Across Train and Test Splits

Why duplicate simulation runs silently inflate test scores

In simulation-based machine learning, a "record" is rarely a single row. One simulation run typically produces a trajectory, a scenario configuration, a random seed, and a set of derived features. If the same run — or a run that is only trivially perturbed — appears in both the training and test partitions, the model can memorize run-specific artifacts instead of learning generalizable dynamics. The result is a test metric that looks excellent during evaluation and collapses in deployment. This case builds a leakage audit that runs before any model training, so the split can be repaired or rejected early.

The audit answers two questions: (1) how many test records have an exact duplicate in the training partition, and (2) how many have a near-duplicate that is close enough to constitute leakage. Both are reported as a leakage rate so the split can be compared against a threshold.

Defining the audit inputs

Assume a simulation dataset exported as a table where each row is one run, with a stable run identifier and a feature vector. A minimal, realistic schema:

Two leakage definitions are used throughout, and they must not be conflated:

• Exact duplicate: identical scenario_hash across splits, or identical feature vectors after rounding. This is unambiguous leakage.
• Near-duplicate: feature vectors whose distance falls below a tolerance. This catches re-runs with a slightly different seed or a minor parameter tweak that did not change the dynamics meaningfully.

Building the audit

The implementation below is self-contained and runnable with pandas and numpy. It generates a small synthetic simulation table with deliberately planted duplicates so the audit output can be verified against known ground truth, then computes both leakage measures.

import numpy as np
import pandas as pd

rng = np.random.default_rng(42)

# --- Synthetic simulation dataset (example data, not real runs) ---
n_train, n_test = 60, 20
feature_cols = [f"f_{i}" for i in range(4)]

train = pd.DataFrame(rng.normal(size=(n_train, 4)), columns=feature_cols)
train["split"] = "train"
train["scenario_hash"] = [f"h{i:04d}" for i in range(n_train)]
train["run_id"] = [f"train_{i:03d}" for i in range(n_train)]

test = pd.DataFrame(rng.normal(size=(n_test, 4)), columns=feature_cols)
test["split"] = "test"
test["scenario_hash"] = [f"t{i:04d}" for i in range(n_test)]
test["run_id"] = [f"test_{i:03d}" for i in range(n_test)]

# Plant leakage: copy 3 train runs into test (exact), and 2 near-duplicates
test.loc[0:2, feature_cols] = train.loc[0:2, feature_cols].values
test.loc[0:2, "scenario_hash"] = train.loc[0:2, "scenario_hash"].values
test.loc[3:4, feature_cols] = train.loc[3:4, feature_cols].values + 1e-4
test.loc[3:4, "scenario_hash"] = ["near_a", "near_b"]

df = pd.concat([train, test], ignore_index=True)

# --- Exact duplicate detection ---
train_hashes = set(df.loc[df.split == "train", "scenario_hash"])
exact_mask = (df.split == "test") & df.scenario_hash.isin(train_hashes)
exact_leaks = df.loc[exact_mask, "run_id"].tolist()

# --- Near-duplicate detection (nearest-neighbor distance) ---
train_feats = df.loc[df.split == "train", feature_cols].to_numpy()
test_feats = df.loc[df.split == "test", feature_cols].to_numpy()

# Euclidean distance from each test run to its closest train run
dists = np.linalg.norm(test_feats[:, None, :] - train_feats[None, :, :], axis=2)
min_dist = dists.min(axis=1)
nearest_train_idx = dists.argmin(axis=1)

TOL = 1e-3
near_mask = min_dist < TOL
near_leaks = df.loc[df.split == "test", "run_id"].to_numpy()[near_mask]

# --- Leakage rates ---
n_test_total = int((df.split == "test").sum())
exact_rate = len(exact_leaks) / n_test_total
near_rate = len(near_leaks) / n_test_total

print(f"test records: {n_test_total}")
print(f"exact duplicates: {len(exact_leaks)} -> rate {exact_rate:.1%}")
print(f"near duplicates (tol={TOL}): {len(near_leaks)} -> rate {near_rate:.1%}")
print("exact leak run_ids:", exact_leaks)
print("near leak run_ids:", near_leaks.tolist())

With the planted ground truth, the expected output is 3 exact duplicates and 2 near-duplicates, giving an exact leakage rate of 15% and a near-duplicate rate of 10% on 20 test records. If the printed counts differ, the planting logic or the tolerance is wrong — not the audit. This is the key verification property: the audit is validated against known leakage before it is trusted on real data.

Reading the audit result

The two rates answer different questions and should be reported separately. Exact duplicates indicate a split-construction bug — the same run was assigned to both partitions, often because the split was made on rows rather than on runs. Near-duplicates indicate a subtler problem: the split was made on runs, but the simulation space was sampled too densely, so distinct runs are effectively the same experiment. The figure below shows how the audit classifies each test run and where the decision boundary sits.

The classification is driven entirely by the nearest-neighbor distance. Exact duplicates collapse to distance zero; near-duplicates sit just below the tolerance; clean runs sit well above it. The tolerance is the only tunable parameter, and it must be chosen relative to the feature scale — a fixed 1e-3 is meaningless if features are not standardized. Standardize features before computing distances, or set the tolerance from the empirical distribution of within-train nearest-neighbor distances.

Choosing a defensible tolerance

A tolerance picked by eye is not defensible in a leakage report. A better approach is to derive it from the training set itself: compute the distribution of nearest-neighbor distances within the training partition, and set the tolerance at a low percentile of that distribution. Runs closer than typical within-train spacing are effectively the same experiment.

from scipy.spatial import cKDTree

tree = cKDTree(train_feats)
within_dists, _ = tree.query(train_feats, k=2)  # k=2: skip self
within_nn = within_dists[:, 1]

TOL = float(np.percentile(within_nn, 5))
print(f"derived tolerance (5th pct of within-train NN): {TOL:.4f}")

test_dists, _ = tree.query(test_feats, k=1)
near_rate = float((test_dists < TOL).mean())
print(f"near-duplicate leakage rate: {near_rate:.1%}")

This makes the tolerance a property of the dataset rather than an arbitrary constant, and it scales automatically when features are standardized. Report the derived tolerance alongside the leakage rate so the number can be reproduced.

Decision rule and deliverable

The audit produces a single verdict per split. A practical rule: reject the split if the exact leakage rate exceeds 0%, or if the near-duplicate rate exceeds 5%. Exact duplicates are always a construction error and have no acceptable threshold. Near-duplicates are tolerated only at a low rate because some feature-space crowding is unavoidable in dense simulation sweeps.

The audit is complete when both rates are computed on the actual split, the tolerance is derived rather than guessed, and the verdict is recorded before any model sees the data. That ordering is what makes the leakage rate meaningful: it describes the split as delivered, not a split that was quietly repaired after a suspiciously high test score.