2. Detect Temporal Leakage in Time-Series Simulation Splits

Why temporal leakage hides in time-series simulation splits

In a time-series simulation, each record carries two clocks: the simulation clock (the simulated timestamp inside the model, e.g. sim_time_s) and the wall-clock at which the run was executed or the data was written (e.g. run_started_at). A random row-wise split ignores both. When you shuffle rows and cut at 80/20, a training window can easily contain timesteps that occur after the earliest test timestep. The model then learns from the future and reports optimistic validation scores that collapse in deployment.

Temporal leakage is not the same as duplicate leakage. Duplicates are identical or near-identical records appearing in both partitions; temporal leakage is an ordering violation — the training set's maximum simulation time exceeds the test set's minimum simulation time, or training windows overlap test windows in clock space. This case builds a small, reproducible audit that compares split boundaries against simulation clock metadata and emits a pass/fail verdict per split.

The audit object: a split manifest plus clock metadata

The audit needs two inputs. First, a split manifest that assigns each simulation run (or each window) to train, val, or test. Second, clock metadata per run: the simulation start time, end time, timestep, and the wall-clock execution time. The check is a boundary comparison, not a model evaluation — it runs in milliseconds and should gate every experiment.

Two boundary rules define the verdict. Rule A (strict temporal order): max(sim_end_s over train) < min(sim_start_s over test). Rule B (no window overlap): no train window [sim_start_s, sim_end_s] intersects any test window. Rule A is the coarse gate; Rule B catches interleaved runs where a single long training run spans the test period even though the aggregate max/min looks fine. A split passes only if both rules hold. The wall-clock field is a cross-check: if run_started_at ordering contradicts simulation ordering, the metadata itself is suspect and the split should be flagged for manual review rather than silently passed.

Implementing the boundary check

The audit is a pure function over the joined manifest and metadata. It computes per-split aggregates, applies Rule A and Rule B, and returns a structured verdict. The code below is self-contained: it defines the example inputs, the check, and the expected output.

from dataclasses import dataclass
from typing import Iterable

@dataclass(frozen=True)
class RunMeta:
    run_id: str
    split: str            # "train" | "val" | "test"
    sim_start_s: float
    sim_end_s: float
    dt_s: float
    run_started_at: str   # ISO 8601 wall-clock

def audit_temporal_split(runs: Iterable[RunMeta]) -> dict:
    runs = list(runs)
    by_split = {}
    for r in runs:
        by_split.setdefault(r.split, []).append(r)

    train = by_split.get("train", [])
    test = by_split.get("test", [])
    if not train or not test:
        return {"status": "INCOMPLETE", "reason": "missing train or test runs"}

    train_max = max(r.sim_end_s for r in train)
    test_min = min(r.sim_start_s for r in test)

    # Rule A: strict temporal order
    rule_a = train_max < test_min

    # Rule B: no train window intersects any test window
    overlaps = []
    for tr in train:
        for te in test:
            if tr.sim_start_s <= te.sim_end_s and te.sim_start_s <= tr.sim_end_s:
                overlaps.append((tr.run_id, te.run_id))

    # Wall-clock cross-check: does execution order agree with sim order?
    train_wall_max = max(r.run_started_at for r in train)
    test_wall_min = min(r.run_started_at for r in test)
    wall_consistent = train_wall_max <= test_wall_min

    status = "PASS" if (rule_a and not overlaps) else "FAIL"
    return {
        "status": status,
        "train_max_s": train_max,
        "test_min_s": test_min,
        "rule_a_order_ok": rule_a,
        "rule_b_overlaps": overlaps,
        "wall_clock_consistent": wall_consistent,
        "leak_margin_s": test_min - train_max,
    }

# --- Example inputs (illustrative, not measured results) ---
runs = [
    RunMeta("sim_0042", "train",    0, 1800, 1.0, "2024-03-11T09:00:00Z"),
    RunMeta("sim_0043", "train", 1800, 3600, 1.0, "2024-03-11T09:05:00Z"),
    RunMeta("sim_0044", "test",  1750, 3550, 1.0, "2024-03-11T09:10:00Z"),
    RunMeta("sim_0045", "train", 3400, 5200, 1.0, "2024-03-11T09:15:00Z"),
]
print(audit_temporal_split(runs))

With these inputs the function returns status="FAIL", rule_a_order_ok=False (train max 5200 s > test min 1750 s), and rule_b_overlaps containing ("sim_0042","sim_0044"), ("sim_0043","sim_0044"), and ("sim_0045","sim_0044"). The leak_margin_s is negative (−3450 s), which is the quantitative signature of leakage: a negative margin means the training horizon extends past the test horizon. A clean split produces a positive margin and an empty overlap list.

Reading the verdict and acting on it

The audit output is designed to be consumed by a CI gate or a pre-training assertion. Interpret it as follows:

• PASS — both rules hold and the margin is positive. Proceed to training; record leak_margin_s in the experiment metadata for traceability.
• FAIL with negative margin — the split is temporally invalid. Rebuild the split by sorting runs on sim_start_s and cutting at a time boundary, not by shuffling rows.
• FAIL with overlaps but positive margin — interleaved windows. Group runs by contiguous simulation segment before splitting, or expand the boundary gap by at least one dt_s.
• INCOMPLETE — a partition is empty; the manifest join likely dropped rows. Verify the run_id join before interpreting any verdict.

Common failure symptoms

When temporal leakage is present but undetected, the usual symptom is a validation metric that is suspiciously high and degrades sharply on a later, truly held-out period. If the audit reports PASS but the model still overfits, check that the manifest's split column was not overwritten after the audit ran, and that sim_end_s reflects the last used timestep rather than the configured simulation duration. Both are metadata errors that make a leaking split look clean.