4. Validate a Surrogate Against a Held-Out High-Fidelity Simulation Set

Why a held-out high-fidelity set is the only honest test

A surrogate model is only as trustworthy as the data it was never allowed to see. In this case we take a trained surrogate — for concreteness, a Gaussian-process or gradient-boosted regressor that maps three input parameters to a scalar structural response — and evaluate it against an independent high-fidelity simulation set that was generated after the surrogate was frozen. The goal is not a single aggregate R² score. It is a stratified error distribution by load level, because a surrogate that is accurate at low load and drifts at high load is dangerous in exactly the regime where engineers care most.

The held-out set must satisfy three conditions before any metric is computed: it was produced by the same solver and mesh family as the training data (so the comparison isolates surrogate error, not solver discrepancy), it was never used for hyperparameter selection, and it spans the load levels we intend to report. If the set was generated in a separate batch, run the near-duplicate check first — a held-out set that secretly contains training points will flatter the surrogate.

Defining the error metrics and the stratification

For each held-out sample i with high-fidelity response y_i and surrogate prediction ŷ_i, compute the signed relative error and its magnitude:

e_i   = (ŷ_i - y_i) / y_i          # signed relative error
ae_i  = |e_i|                        # absolute relative error
ape_i = ae_i * 100                   # absolute percentage error

Report the signed error because it reveals bias (systematic over- or under-prediction), and the absolute error because it reveals spread. A surrogate with mean signed error near zero but a heavy absolute-error tail is still unreliable. Stratify by load level using the load parameter recorded in the simulation metadata, not by binning the response — binning on the output would leak information and distort the picture.

Use fixed, physically meaningful load bins defined before looking at errors, for example Low (0–33% of rated load), Mid (33–66%), and High (66–100%). Fixed bins keep the report comparable across surrogate versions and prevent the temptation to re-bin until the numbers look good.

Reference implementation

The following script is self-contained: it builds a small synthetic high-fidelity set with a known nonlinear response, trains a surrogate on a separate training split, then evaluates on the held-out set and prints the stratified table. The synthetic response is a stand-in for solver output; replace high_fidelity_response with your actual simulation results and the rest of the pipeline is unchanged.

import numpy as np
import pandas as pd
from sklearn.ensemble import GradientBoostingRegressor

rng = np.random.default_rng(7)

def high_fidelity_response(load, stiffness, thickness):
    # Stand-in for FEA output: nonlinear in load, mild coupling.
    return (1.0 + 0.9 * load**1.6) * (stiffness**-0.5) * (1.0 + 0.15 * thickness)

def make_set(n, seed):
    r = np.random.default_rng(seed)
    load       = r.uniform(0.0, 1.0, n)      # normalized load level
    stiffness  = r.uniform(0.8, 1.2, n)
    thickness  = r.uniform(0.9, 1.1, n)
    X = np.column_stack([load, stiffness, thickness])
    y = high_fidelity_response(load, stiffness, thickness)
    return X, y, load

# Training split (surrogate sees this) and independent held-out set.
X_tr, y_tr, _      = make_set(400, seed=1)
X_ho, y_ho, load_ho = make_set(120, seed=99)   # different seed => independent

surrogate = GradientBoostingRegressor(
    n_estimators=300, max_depth=3, learning_rate=0.05, random_state=0
).fit(X_tr, y_tr)

y_hat = surrogate.predict(X_ho)
rel_err = (y_hat - y_ho) / y_ho

# Fixed load bins defined before inspection.
bins   = [0.0, 0.33, 0.66, 1.0001]
labels = ["Low", "Mid", "High"]
stratum = pd.cut(load_ho, bins=bins, labels=labels, right=False)

df = pd.DataFrame({
    "load": load_ho,
    "stratum": stratum,
    "rel_err": rel_err,
    "ape": np.abs(rel_err) * 100.0,
})

summary = df.groupby("stratum", observed=True).agg(
    n=("ape", "size"),
    mean_signed_pct=("rel_err", lambda s: s.mean() * 100.0),
    mape=("ape", "mean"),
    p95_ape=("ape", lambda s: np.percentile(s, 95)),
    max_ape=("ape", "max"),
).round(3)

print(summary)
print("\nOverall MAPE: %.3f%%" % df["ape"].mean())
print("Overall P95 APE: %.3f%%" % np.percentile(df["ape"], 95))

Running this produces a table with one row per load stratum. The exact numbers depend on the random seeds and the surrogate configuration, so treat any output as an illustrative result of this pipeline rather than a benchmark. What matters is the shape of the table: a well-behaved surrogate shows MAPE and P95 APE that grow modestly from Low to High, while a surrogate that has not learned the load nonlinearity shows a sharp jump in the High stratum.

Reading the stratified error distribution

The table alone is not a diagnosis. Interpret it against three patterns:

  • Uniformly low error across strata — the surrogate generalizes; proceed to deployment checks.
  • Rising MAPE with load, but mean signed error near zero — the surrogate captures the trend but loses precision at high load; consider adding high-load training samples or a load-weighted loss.
  • Mean signed error drifting negative or positive with load — systematic bias, typically from a training set that under-represents the high-load regime or from a model that cannot represent the load nonlinearity. This is the most dangerous pattern because it is invisible in an aggregate R².

Always report the sample count per stratum. A High-load stratum with only six points produces a P95 APE that is essentially the maximum of six numbers and should not be quoted as a percentile. If a stratum has fewer than ~20 samples, report min/median/max instead of P95 and flag the small sample size.

Plot the signed relative error against load as a scatter with a zero line. Bias that grows with load is far easier to see in that plot than in any summary table, and it directly motivates the next action: targeted sampling or a model-form change.
Do not tune the surrogate on the held-out set. If the stratified errors are unacceptable, the correct response is to revise the training data or model and then generate a new held-out set. Reusing the same held-out set after tuning silently converts it into a validation set and reintroduces the optimistic bias this case is designed to expose.
Held-out HF set independent batch never used for tuning X_ho, y_ho, load_ho n = 120 Frozen surrogate GradientBoosting trained on X_tr, y_tr predict(X_ho) → ŷ Error metrics e = (ŷ − y)/y APE = |e|·100 signed + magnitude Stratify by load level (fixed bins, defined before inspection) Low 0–33% · Mid 33–66% · High 66–100% Low load MAPE low, bias ≈ 0 P95 APE small n sufficient Mid load MAPE moderate bias near zero spread acceptable High load MAPE rises sharply signed bias drifts → targeted resampling

Reporting checklist

Deliver the stratified table with these columns so a reviewer can audit the claim: stratum, n, mean_signed_pct, MAPE, P95_APE, max_APE. State the bin edges, the held-out set size, and the surrogate version hash. If any stratum has n < 20, replace P95 with min/median/max and annotate the small sample. The acceptance criterion should be set before evaluation — for example, "MAPE ≤ 5% and |mean signed error| ≤ 2% in every stratum" — and the surrogate either meets it or does not. A surrogate that passes overall but fails the High stratum is not accepted; it is flagged for targeted resampling in the high-load regime, which is precisely the diagnostic this case is built to surface.

← Previous 📖 Contents Next →