4. Calibrating OOD Thresholds Using LS-DYNA Validation Runs

Why Threshold Calibration Needs a Labeled Validation Set

An OOD score is only useful once a decision boundary is attached to it. In the LS-DYNA fallback pipeline, that boundary is a scalar threshold τ: when the OOD score s(x) of a run exceeds τ, the orchestrator abandons the fast nominal solver configuration and re-runs the deck with a conservative configuration (smaller timestep, tighter contact controls, reduced mass scaling). Choosing τ by intuition produces one of two failure modes. Set it too low and every benign variation — a slightly different mesh density, a legitimate material scatter — triggers an expensive fallback, wasting solver hours. Set it too high and genuinely anomalous decks (bad contact definitions, unstable hourglass modes, corrupted initial velocities) run to completion and return garbage.

Calibration therefore requires a validation set of LS-DYNA runs whose ground truth is known: a group of nominal decks that must not trigger fallback, and a group of anomalous decks that must trigger it. The task in this case is to sweep candidate thresholds across that set, measure both error types, and select the operating point that best balances them.

Defining the Two Error Costs

Two error rates drive the decision, and they are not symmetric in cost:

• False fallback rate (FFR) — fraction of nominal runs whose score exceeds τ. Each occurrence costs a full conservative re-run, so it is a throughput penalty.
• Missed detection rate (MDR) — fraction of anomalous runs whose score stays at or below τ. Each occurrence lets a bad simulation pass as valid, which is a correctness penalty and usually the more expensive one.

Because the two costs differ, a plain "minimize total errors" objective is often wrong. A weighted cost function makes the trade-off explicit:

J(τ) = C_fallback · FFR(τ) + C_miss · MDR(τ)

where C_fallback is the cost of one unnecessary conservative re-run and C_miss is the cost of accepting one invalid result. For a crashworthiness study where a missed anomaly could invalidate a certification run, C_miss is typically set 5–20× larger than C_fallback.

Building the Validation Set

The validation set must be small enough to run overnight but broad enough to cover the score distribution. A workable composition for a single vehicle crash model is:

Each run produces a d3plot file; the feature extractor from the earlier stage reduces it to a feature vector, and the OOD scorer emits a single scalar s(x). The calibration step consumes only the pairs (s(x), label), so it is independent of how the score was computed.

Sweeping Thresholds and Reading the Trade-off

With scores and labels in hand, the sweep is mechanical. Sort all scores, place candidate thresholds at the midpoints between consecutive distinct scores, and for each candidate compute FFR and MDR. The result is a pair of monotone curves that cross the cost function at the optimum.

import numpy as np

def calibrate(scores, labels, c_fallback=1.0, c_miss=10.0):
    scores = np.asarray(scores, dtype=float)
    labels = np.asarray(labels)          # 1 = anomalous, 0 = nominal
    order = np.argsort(scores)
    s_sorted = scores[order]
    # candidate thresholds at midpoints between distinct scores
    uniq = np.unique(s_sorted)
    cands = (uniq[:-1] + uniq[1:]) / 2.0
    cands = np.concatenate(([uniq[0] - 1e-9], cands, [uniq[-1] + 1e-9]))

    n_nom = np.sum(labels == 0)
    n_anom = np.sum(labels == 1)
    rows = []
    for tau in cands:
        pred = scores > tau
        ffr = np.sum((labels == 0) & pred) / n_nom
        mdr = np.sum((labels == 1) & ~pred) / n_anom
        rows.append((tau, ffr, mdr, c_fallback * ffr + c_miss * mdr))
    return rows

# example: scores from the validation set, labels 0/1
scores = [0.12, 0.18, 0.21, 0.25, 0.31, 0.34, 0.38, 0.41, 0.44, 0.47,
          0.52, 0.55, 0.58, 0.61, 0.66, 0.71, 0.74, 0.79, 0.83, 0.88,
          0.91, 0.95, 1.02, 1.08, 1.15, 1.21, 1.30, 1.42, 1.55, 1.71]
labels = [0]*20 + [1]*10

rows = calibrate(scores, labels, c_fallback=1.0, c_miss=10.0)
best = min(rows, key=lambda r: r[3])
print(f"tau*={best[0]:.3f}  FFR={best[1]:.2f}  MDR={best[2]:.2f}  J={best[3]:.2f}")

Running this on the illustrative scores above yields τ* ≈ 0.90, with FFR ≈ 0.05 and MDR ≈ 0.10 under a 10:1 miss-to-fallback cost ratio. The exact numbers depend on the score distribution; what matters is the procedure and the cost weighting, not the specific value.

Visualizing the Operating Point

The figure below shows the score distributions of the two groups and how the chosen threshold sits between them, together with the resulting error regions. This is the single most useful diagnostic during calibration: it reveals whether the groups are separable at all, or whether the scorer itself needs revisiting before any threshold can help.

Acceptance Criteria and Failure Symptoms

A calibrated threshold is accepted only if it meets pre-agreed limits on the held-out set. Typical gates for a crash model are FFR ≤ 0.10 and MDR ≤ 0.05 under the chosen cost ratio. If the sweep cannot reach both gates simultaneously, the problem is usually upstream, not in the threshold:

• Distributions overlap heavily — the feature set does not separate the classes; revisit the d3plot features (contact force peaks, kinetic energy decay) rather than lowering the threshold.
• FFR spikes on one nominal run — a single outlier deck is inflating the score; inspect whether that run has a legitimate but extreme parameter (e.g., very coarse mesh) that should be added to the nominal training distribution.
• MDR stays high even at low τ — the anomalous decks are producing scores indistinguishable from nominal, meaning the scorer is blind to that anomaly class; add a dedicated feature or a second detector.

Once the threshold passes the gates, record τ*, the cost ratio, the validation-set composition, and the held-out error rates together. That record is what makes the fallback trigger auditable when a later run is questioned.