4. Data generation from CAE solvers

A surrogate model is only as good as the dataset that trains it. In a CAE-driven design optimization workflow, that dataset is produced by repeatedly perturbing design variables, re-running the solver, and harvesting scalar or field outputs. The engineering challenge is not running one simulation — it is running hundreds or thousands of them reliably, reproducibly, and with enough metadata that a failed or suspicious run can be identified and repaired rather than silently poisoning the training set. This section covers the practical machinery: how to automate solver invocation, how to couple the solver to a design-of-experiments (DoE) driver, how to execute batches efficiently, and how to detect, classify, and handle simulation failures.

From a Single Run to a Parameterized Pipeline

The foundation of automation is a parameterized case: a template model in which every design variable is exposed as a named input, and every quantity of interest (QoI) is exposed as a named output. For a structural bracket example, the design variables might be thickness_web, thickness_flange, fillet_radius, and rib_height; the QoIs might be max_von_mises, mass, and first_eigenfrequency. The template must be driven entirely by a machine-readable input file — a solver deck, a JSON parameter block, or a Python API call — so that no human edits are needed between runs.

A robust pipeline separates four concerns: sampling (generating the DoE matrix), case instantiation (writing per-run input decks into isolated working directories), execution (launching the solver and capturing logs), and extraction (parsing QoIs into a tabular record). Keeping these stages decoupled means a failed run can be re-instantiated and re-executed without regenerating the whole DoE, and extraction logic can be tested independently of solver runtime.

runs/
  run_0001/
    case.inp          # instantiated solver deck
    params.json       # design variables for this run
    solver.log        # stdout/stderr capture
    results.json      # extracted QoIs + status
  run_0002/
    ...
manifest.csv        # run_id, status, wall_time, QoIs

Solver Coupling Strategies

Coupling the DoE driver to the solver can be done at three levels of intimacy, and the right choice depends on solver licensing, API availability, and throughput requirements.

• File-based (black-box) coupling. The driver writes an input deck, calls the solver executable as a subprocess, waits for exit, and parses output files. This is the most portable approach and works with any solver that accepts a text deck. It is also the easiest to parallelize because each run is a fully isolated process.
• Script/API coupling. Many solvers expose a Python or COM API (for example, a journaling interface or a headless batch mode). This avoids file parsing fragility and allows in-memory parameter injection, but ties the pipeline to a specific solver version.
• In-process coupling. The solver is linked as a library and called directly. This minimizes overhead but complicates parallel execution and fault isolation — a solver crash can take down the driver.

For most surrogate-generation campaigns, file-based coupling with a thin Python wrapper is the pragmatic default: it is debuggable, restartable, and trivially distributable across a compute cluster. The wrapper's job is to enforce a strict contract — given a params.json, produce a results.json with a status field, or fail loudly with a captured log.

Batch Execution and Throughput

Batch execution is where wall-clock time is won or lost. Two levers matter: parallelism and solver-internal threading. Running N cases concurrently with one thread each usually beats running them serially with N threads, because CAE solvers scale sub-linearly beyond a few cores and because independent cases have no synchronization cost. A typical configuration on a 32-core node is 8 concurrent cases × 4 threads each, tuned by measuring actual throughput rather than assuming it.

Batch drivers should also implement: a per-run timeout (a hung solve must not block the queue), a retry policy for transient failures (license checkout, scratch-disk contention), and checkpointing so that a crashed batch resumes from the manifest rather than restarting from run 1. The manifest is the single source of truth — every run's status, wall time, and QoIs live there, and downstream training reads only from it.

Handling Simulation Failures

In a campaign of 500 runs, expect 5–15% to fail or produce suspect results. Failures fall into recognizable classes, and each demands a different response. The figure below maps the failure taxonomy to detection signals and remediation actions.

Hard failures are unambiguous: the solver exits non-zero, diverges, or produces no output file. Common causes include an infeasible geometry (self-intersecting fillet at a large radius), a mesh that fails to generate at an extreme parameter value, or a transient license checkout failure. The first two are design-space problems — the sampled point is physically or numerically invalid — and should be recorded as such, not retried indefinitely. The third is infrastructure and should be retried with backoff.

Soft failures are more insidious: the solver exits cleanly and writes a number, but the number is wrong. A stress value below yield but ten times higher than neighboring design points, a mass that is negative, or an eigenfrequency of zero all indicate that the run "succeeded" numerically while failing physically. These must be caught by automated sanity checks before the record enters the training set. A practical rule set for the bracket example:

Making Failures Informative

The most valuable habit in data generation is to treat every failure as a labeled data point about the design space, not as noise to be discarded. A run that failed because the fillet radius exceeded the local web width tells you where the feasible boundary lies — information a surrogate trained only on successful runs will never learn. Record failures in the manifest with a status field (ok, hard_fail, soft_fail) and a reason string, and consider training a classifier on the failure labels alongside the regression surrogate. This "feasibility surrogate" can then steer the optimizer away from infeasible regions before expensive solver calls are spent.

Finally, reproducibility closes the loop. Each run directory should capture the solver version, the exact input deck, the parameter values, and the wall time. When a QoI later looks anomalous during surrogate validation, the original run can be re-executed and compared byte-for-byte. Without this provenance, debugging a bad prediction becomes guesswork. With it, the data-generation pipeline becomes an auditable, restartable, and self-documenting component of the optimization workflow — which is exactly what a surrogate model needs beneath it.