4. Build a Leakage-Aware Splitter for Parametric Simulation Sweeps

Why parameter families, not individual runs, define the split unit

In a parametric simulation sweep, each run is not an independent sample. Runs that share a parameter family — the same scenario template, the same agent population, the same environment configuration, or the same random seed lineage — are statistically correlated. If you split at the run level, a run with vehicle_count=40, spawn_rate=0.8 may land in training while its near-twin vehicle_count=42, spawn_rate=0.8 lands in test. The model then sees the test distribution during training, and your generalization estimate becomes optimistic. The correct split unit is therefore the parameter family: every run belonging to a family must be assigned to exactly one side of the boundary.

This chapter implements a leakage-aware splitter that (1) derives a family key from the sweep configuration, (2) groups runs by that key, and (3) assigns whole groups to train/test so that no family crosses the boundary. We also add a verification pass that fails loudly if any family leaks.

Defining the family key

The family key must capture the parameters that induce correlation. A practical rule: include every parameter that is held constant within a correlated block and exclude the parameter that is being swept for coverage. For a traffic simulation sweep, a defensible key is the tuple of scenario template, environment map, and seed lineage — while the swept parameters (vehicle count, spawn rate) vary within a family and should not fragment it.

import hashlib
import json
from collections import defaultdict

# Parameters that define a correlated family. Runs sharing these
# values are treated as one indivisible unit for splitting.
FAMILY_FIELDS = ("scenario_template", "environment_map", "seed_lineage")

def family_key(run: dict) -> str:
    """Stable, order-independent key for a run's parameter family."""
    payload = {k: run[k] for k in FAMILY_FIELDS}
    blob = json.dumps(payload, sort_keys=True, separators=(",", ":"))
    return hashlib.sha256(blob.encode("utf-8")).hexdigest()[:16]

Hashing keeps the key compact and avoids delimiter collisions when field values contain separators. Sorting keys makes the key independent of dict insertion order, which matters when runs are loaded from different sources.

Grouping and assigning whole families

Once every run has a family key, we group runs and assign groups — never individual runs — to train or test. A deterministic hash of the family key gives a stable assignment that is reproducible across machines and does not depend on row order.

def split_by_family(runs, test_fraction=0.25, salt="v1"):
    """Assign whole parameter families to train/test.

    Returns (train_runs, test_runs, report).
    """
    groups = defaultdict(list)
    for run in runs:
        groups[family_key(run)].append(run)

    train, test = [], []
    for key, members in groups.items():
        # Deterministic bucket in [0, 1) from the family key.
        digest = hashlib.sha256(f"{salt}:{key}".encode()).hexdigest()
        bucket = int(digest[:8], 16) / 0xFFFFFFFF
        (test if bucket < test_fraction else train).extend(members)

    report = {
        "n_runs": len(runs),
        "n_families": len(groups),
        "train_runs": len(train),
        "test_runs": len(test),
        "train_families": len({family_key(r) for r in train}),
        "test_families": len({family_key(r) for r in test}),
    }
    return train, test, report

Because assignment is per family, the realized test fraction will drift from test_fraction when families have unequal sizes. That is expected and correct: preserving family integrity outranks hitting an exact ratio. The report exposes the realized counts so you can confirm the drift is acceptable.

Verifying that no family crosses the boundary

The splitter is only trustworthy if it can prove its own invariant. The check below recomputes family keys on the outputs and asserts disjointness. Run it every time the sweep configuration changes.

def assert_no_family_leakage(train, test):
    train_keys = {family_key(r) for r in train}
    test_keys = {family_key(r) for r in test}
    overlap = train_keys & test_keys
    if overlap:
        raise AssertionError(
            f"Leakage: {len(overlap)} parameter families appear in both "
            f"splits, e.g. {sorted(overlap)[:3]}"
        )
    return True

Worked example with a small sweep

Consider six runs across two scenario templates and two seed lineages. The table shows the family key inputs and the resulting assignment with test_fraction=0.25.

Runs r01 and r02 differ only in vehicle_count, so they share family F1 and stay together. The same holds for F2 and F3. The realized split is 4 train / 2 test runs across 2 train families and 1 test family — a 33% test fraction rather than 25%, which is the expected drift from family granularity. The leakage assertion passes because F1, F2, and F3 are each wholly on one side.

Failure symptoms and troubleshooting

• Assertion fires after a config change. A new field was added to the sweep that should be part of FAMILY_FIELDS but was omitted, so two correlated blocks collapsed into one key or split apart. Re-audit which parameters are held constant within a correlated block.
• Test fraction far from target. One or two very large families dominate the sweep. Either accept the drift, or stratify by family size before hashing — but never split a family to hit the ratio.
• Different splits on different machines. The salt or the field set differs, or run dicts carry extra keys that leak into the hash. Pin salt in config and hash only FAMILY_FIELDS.
• Families look identical but keys differ. Whitespace or case differences in string fields. Normalize values (strip, lowercase) before hashing.