3. Measure Entity Overlap Between Train and Test Simulation Scenarios

Why entity overlap matters in simulation splits

In simulation datasets, a "record" is rarely an isolated sample. Each run is generated from a scenario that bundles several entities: the agents being simulated, the environment or map they operate in, and the random seed that drives stochastic behavior. If the same agent, environment, or seed appears in both the training and test partitions, the test set is no longer an independent estimate of generalization — the model has effectively seen that entity before. This is a form of leakage that survives even when individual records are not exact duplicates and even when timestamps are cleanly separated.

This case builds a small, self-contained audit that measures entity overlap between splits. For each entity type we compute the fraction of test-side entities that also occur on the training side, then flag any split whose overlap exceeds a configurable threshold. The output is a per-entity-type overlap report plus a pass/fail verdict per split.

Defining the overlap metric

Let T be the set of entities of a given type present in the training split and E the corresponding set in the evaluation (test) split. The overlap fraction we report is the contamination of the test side:

overlap_fraction = |T ∩ E| / |E|

This answers the practical question: "Of the entities the model is evaluated on, what share did it already see during training?" A value of 0 means the test entities are fully disjoint from training; a value of 1 means every test entity was also in training. We compute this separately for agents, environments, and seeds, because the three leak in different ways and have different tolerances. Seeds, for instance, are often shared across many runs by design, so a small seed overlap may be acceptable while any agent overlap is usually a red flag.

Reference implementation

The audit below is complete and runnable with only the Python standard library. It reads a list of run records, each carrying its split label and the entity identifiers used to generate it, then produces the overlap report and flags.

from collections import defaultdict

# Each run: split label + the entities that generated it.
runs = [
    {"run_id": "r01", "split": "train", "agent": "agv-A", "env": "warehouse-1", "seed": 11},
    {"run_id": "r02", "split": "train", "agent": "agv-A", "env": "warehouse-1", "seed": 12},
    {"run_id": "r03", "split": "train", "agent": "agv-B", "env": "warehouse-2", "seed": 13},
    {"run_id": "r04", "split": "train", "agent": "agv-C", "env": "warehouse-2", "seed": 14},
    {"run_id": "r05", "split": "test",  "agent": "agv-A", "env": "warehouse-3", "seed": 11},
    {"run_id": "r06", "split": "test",  "agent": "agv-D", "env": "warehouse-3", "seed": 21},
    {"run_id": "r07", "split": "test",  "agent": "agv-E", "env": "warehouse-4", "seed": 22},
    {"run_id": "r08", "split": "test",  "agent": "agv-F", "env": "warehouse-4", "seed": 23},
]

ENTITY_FIELDS = ["agent", "env", "seed"]
THRESHOLDS = {"agent": 0.0, "env": 0.0, "seed": 0.25}

def collect_entities(records, split, field):
    return {r[field] for r in records if r["split"] == split}

def overlap_report(records, fields, thresholds):
    report = {}
    for field in fields:
        train = collect_entities(records, "train", field)
        test = collect_entities(records, "test", field)
        shared = train & test
        frac = len(shared) / len(test) if test else 0.0
        report[field] = {
            "train_count": len(train),
            "test_count": len(test),
            "shared": sorted(shared),
            "overlap_fraction": round(frac, 3),
            "threshold": thresholds[field],
            "flagged": frac > thresholds[field],
        }
    return report

report = overlap_report(runs, ENTITY_FIELDS, THRESHOLDS)

for field, r in report.items():
    status = "FLAG" if r["flagged"] else "ok"
    print(f"{field:6s} train={r['train_count']} test={r['test_count']} "
          f"shared={r['shared']} overlap={r['overlap_fraction']:.3f} "
          f"(thr={r['threshold']}) -> {status}")

any_flag = any(r["flagged"] for r in report.values())
print("SPLIT VERDICT:", "REJECT" if any_flag else "ACCEPT")

Running this against the sample data produces the following expected output (these are the values implied by the input above, not a claim about any external dataset):

agent  train=3 test=4 shared=['agv-A'] overlap=0.250 (thr=0.0) -> FLAG
env    train=2 test=2 shared=[] overlap=0.000 (thr=0.0) -> ok
seed   train=4 test=4 shared=[11] overlap=0.250 (thr=0.25) -> ok
SPLIT VERDICT: REJECT

The verdict is REJECT because agv-A appears in both splits. Note that the seed overlap of 0.25 sits exactly at its threshold and is not flagged, since the comparison is strict (frac > threshold). This is a deliberate design choice: seeds are frequently reused across runs, so a tolerance is reasonable, whereas agent reuse is treated as zero-tolerance.

Reading the overlap structure

The diagram below shows how the sample runs map onto the three entity dimensions and where the shared entities cross the split boundary. It makes the leakage visible at a glance: agv-A and seed 11 are the only entities that appear on both sides, and they are exactly what drives the flags.

Choosing thresholds and acting on flags

The threshold table is the policy layer of the audit. The values below are a reasonable starting point for simulation workloads; adjust them to your domain, but keep the reasoning explicit so reviewers can see why a given overlap was tolerated.

When a split is flagged, the fix is almost always to change the grouping key used by the splitter, not to delete records. For an agent flag, group runs by agent and assign whole groups to one side. For an environment flag, group by environment. The audit is cheap to re-run after re-splitting, so treat it as a gate: run it, read the verdict, re-split if rejected, and re-run until the verdict is ACCEPT.

Verification and common failure modes

To confirm the audit behaves correctly, check three things against the sample data. First, the agent overlap must be exactly 0.25 (one shared agent out of four test agents) and flagged. Second, the environment overlap must be 0.0 because the two environment sets are disjoint. Third, the seed overlap must be 0.25 and not flagged, confirming the strict comparison at the threshold boundary. If any of these differ, the most likely causes are:

• Empty test set: the guard if test else 0.0 prevents a division-by-zero, but an empty test split should be treated as an error upstream, not silently scored as zero overlap.
• Type mismatch on seeds: seeds read from CSV arrive as strings while in-memory seeds are integers, so 11 and "11" never intersect. Normalize types before building the sets.
• Missing entity field: a run lacking an env key raises KeyError; validate the schema before auditing so a malformed record does not masquerade as a clean split.

Once the report is clean, persist it alongside the split manifest so the overlap measurement travels with the dataset version and can be re-checked whenever the split is regenerated.