4. Labeling and ground truth for numerical failures

Supervised detection of numerical failures in crash simulation is only as good as the labels behind it. A model trained to flag hourglassing, negative Jacobians, or contact blow-ups will inherit every inconsistency in the annotation protocol that produced its training set. This section covers three tightly coupled problems: how to define and apply a manual annotation protocol, how to locate the exact onset frame of a failure, and how to keep the resulting dataset usable when failures are rare relative to healthy runs.

What Counts as a Label

In crash simulation, a "numerical failure" is not a single event but a family of them. Before any annotation begins, the taxonomy must be frozen and written down. A practical working set for LS-DYNA-style explicit crash models includes:

• Negative volume / negative Jacobian — an element inverts, typically reported in d3hsp as negative volume with an element ID and cycle number.
• Hourglassing — energy in hourglass modes exceeds a threshold fraction of internal energy, or the hourglass energy ratio in glstat crosses a limit.
• Contact instability — penetration depth beyond a tolerance, or contact energy growth that is not physically explained by the impact.
• Time step collapse — the global time step in glstat drops below a floor because of a single distorted element.
• Solver termination — explicit error termination with a documented cause code.

Each label needs a machine-checkable definition, not a verbal one. "Hourglassing" is not a label; "hourglass energy / internal energy > 0.10 sustained for ≥ 5 consecutive output states" is. The threshold values are model-dependent and must be recorded per dataset, because a 0.10 ratio is alarming in a full-vehicle frontal impact and unremarkable in a foam-dominated side impact.

Manual Annotation Protocol

Manual annotation is expensive, so the protocol should minimize the number of decisions an annotator makes while maximizing reproducibility. A workable protocol has four stages.

1. Pre-screening. Run an automated rule pass over glstat, matsum, and d3hsp to produce candidate intervals. This does not create labels; it only narrows the frames an annotator must inspect.
2. Frame inspection. The annotator scrubs the candidate interval in the post-processor, viewing the relevant field (effective plastic strain, hourglass energy density, contact penetration) and records the first frame where the failure criterion is met.
3. Onset marking. The annotator marks the onset frame, not the frame where the failure became obvious. This distinction is covered below.
4. Adjudication. A second annotator reviews a random 10–15% sample. Disagreements are resolved by a fixed tie-break rule, and the resolution is logged so the same case is not re-litigated.

Failure Onset Detection

The onset frame is the earliest state at which the failure condition is satisfied, and it is almost never the frame where the failure is visually dramatic. In an explicit crash run with a 1 µs stable time step and output every 1 ms, a negative-volume element may invert at t = 18.4 ms but only produce a visible mesh distortion at t = 21 ms. Labeling the visible frame teaches the model to detect consequences, not causes.

Three onset definitions are common, and the choice must be consistent across the dataset:

For solver-reported failures, the cycle number in d3hsp is authoritative and should be converted to simulation time using the recorded time step. For energy-based failures, first-sustained with N = 3 to 5 output states is usually the most stable choice, because a single output frame can spike from a transient contact correction that resolves itself.

A subtle but important point: onset detection must be done on the same output cadence the model will see at inference. If the model is trained on 1 ms output but the production pipeline writes every 0.5 ms, the onset frame indices will not correspond, and the model will appear to fire late.

Class Imbalance

In a realistic crash simulation dataset, healthy runs dominate. A dataset of 5,000 runs might contain 300 with a numerical failure, and of those, perhaps 40 with a specific failure mode such as negative volume. Naive training on this distribution produces a model that predicts "healthy" for everything and still achieves 94% accuracy — a useless result.

Imbalance must be handled at three levels, and doing only one is usually insufficient.

• Sampling. At the window level, oversample failure windows and undersample healthy windows so the effective ratio is between 1:3 and 1:10. Pure 1:1 oversampling tends to overfit the few failure examples.
• Loss weighting. Apply class weights inversely proportional to frequency, or use focal loss to down-weight easy healthy windows. This is complementary to sampling, not a replacement.
• Evaluation. Report per-class recall and precision, and use PR-AUC rather than ROC-AUC. With heavy imbalance, ROC-AUC stays deceptively high because the false-positive rate is diluted by the large healthy class.

Putting It Together

The diagram below shows how a raw simulation run moves through pre-screening, onset marking, and windowed sampling into a balanced training set. The key idea is that the label is attached to a specific frame interval, not to the run as a whole, and that balancing happens after the run-level train/validation split.

Practical Notes on Thresholds and Reproducibility

Thresholds should be stored in a versioned configuration file, not hard-coded in the annotation script. A minimal example for an hourglass-energy rule:

failure_rules:
  hourglass:
    field: glstat.hourglass_energy / glstat.internal_energy
    threshold: 0.10
    sustain_frames: 5
    onset_definition: first_sustained
  negative_volume:
    source: d3hsp
    pattern: "negative volume"
    onset_definition: solver_reported
  contact_penetration:
    field: glstat.max_penetration
    threshold: 2.0   # mm, model-dependent
    sustain_frames: 3
    onset_definition: first_sustained

When the threshold changes, the dataset version must change with it. A model trained on threshold 0.10 and evaluated against labels generated at 0.15 will show a systematic offset in onset frame that looks like a model error but is a labeling artifact.

Finally, keep a small "golden set" of 50–100 runs that are fully hand-annotated and never used for training. Every time the protocol or thresholds change, re-run the automated pipeline against the golden set and report agreement. If agreement drops, the change is not ready to propagate to the full dataset. This single practice catches most of the silent label drift that otherwise accumulates across a long annotation campaign.