4. Automating CAE solver runs

Design space exploration multiplies a single analysis into dozens or hundreds of solver runs. A structural study sweeping five thickness values across three load cases, a thermal study varying conductivity and film coefficients, or a CFD study perturbing inlet velocity and turbulence intensity all share the same bottleneck: the solver itself is not the slow part — the manual work around it is. This section covers the automation layer that turns a parameterized model into a managed batch of solver jobs, with reliable input deck generation, submission, and monitoring for structural, thermal, and fluid analyses.

From Parameter Set to Input Deck

Most CAE solvers consume a text input deck: Abaqus .inp, ANSYS APDL .dat, Nastran .bdf, Fluent journal files, or OpenFOAM dictionaries. The automation pattern is templating: keep a master deck with placeholder tokens, then render one concrete deck per design point.

A minimal, solver-agnostic approach uses Python string substitution over a template file:

from pathlib import Path
from string import Template

TEMPLATE = Template(Path("plate_master.inp").read_text())

def render_deck(run_dir: Path, params: dict) -> Path:
    run_dir.mkdir(parents=True, exist_ok=True)
    deck = TEMPLATE.substitute(
        THICKNESS=f"{params['thickness']:.4f}",
        YOUNGS=f"{params['youngs']:.1f}",
        LOAD=f"{params['load']:.2f}",
        JOBNAME=params["jobname"],
    )
    out = run_dir / f"{params['jobname']}.inp"
    out.write_text(deck)
    return out

Three rules keep deck generation trustworthy. First, format every numeric value explicitly — f"{x:.4f}" rather than raw str(x) — because solvers are sensitive to exponent notation and locale-dependent decimal separators. Second, never mutate the master template in place; always write into a per-run directory so a failed run leaves an auditable artifact. Third, validate the rendered deck before submission: check that no $ token remains, that required keywords are present, and that the file is non-empty.

Batch Execution and Job Submission

Once decks exist, submission is a matter of launching the solver with the right command line and capturing its exit status. For local batch execution, subprocess.run with an explicit working directory and a timeout is sufficient:

import subprocess

def run_abaqus(deck: Path, cpus: int = 4, timeout_s: int = 3600):
    cmd = ["abaqus", f"job={deck.stem}", f"cpus={cpus}",
           "interactive", "scratch=./scratch"]
    proc = subprocess.run(
        cmd, cwd=deck.parent, capture_output=True,
        text=True, timeout=timeout_s,
    )
    return proc.returncode, proc.stdout, proc.stderr

For thermal and fluid solvers the pattern is identical but the invocation differs: Fluent typically runs in batch mode via fluent 3ddp -g -t4 -i run.jou, while OpenFOAM solvers are launched through foamRun or a legacy solver binary after sourcing the environment. The key discipline is the same: one process per design point, isolated working directory, captured stdout/stderr, and a recorded exit code.

On an HPC cluster, replace direct execution with a scheduler submission. Generate a job script per run and submit it, then poll the scheduler for state:

#!/bin/bash
#SBATCH --job-name=plate_t3.0
#SBATCH --nodes=1
#SBATCH --ntasks=8
#SBATCH --time=02:00:00
#SBATCH --output=slurm-%j.out

module load abaqus/2023
cd $SLURM_SUBMIT_DIR
abaqus job=plate_t3.0 cpus=8 interactive

Submission from Python uses sbatch and parses the returned job ID; monitoring uses squeue -j <jobid> -h -o %T or sacct for completed jobs. The same abstraction works for PBS (qsub/qstat) and LSF (bsub/bjobs).

Monitoring, Failure Detection, and Result Harvesting

A batch of 200 jobs will contain failures. Monitoring must distinguish three states: running, completed successfully, and failed — and for the last, capture enough context to diagnose without rerunning.

For local runs, the exit code is the primary signal, but solvers sometimes return zero while emitting warnings that invalidate results. Parse the solver log for convergence and error markers. Abaqus writes THE ANALYSIS HAS COMPLETED SUCCESSFULLY to the .sta file; Fluent reports residuals and a final Calculation complete; OpenFOAM writes FOAM FATAL ERROR on failure. A robust monitor checks both the process status and the log content.

The following table summarizes the observable signals per solver family:

Harvesting results means extracting scalar objectives from each run and assembling them into a design-point table. For structural runs, parse the .dat or query the .odb via the solver's Python API; for thermal, read nodal temperatures; for fluid, extract pressure drop and mass-flow-weighted outlet temperature. Store each row keyed by the design-point ID so it joins cleanly with the parameter table.

Concurrency and Resource Discipline

Running jobs in parallel is where throughput is won, but naive ThreadPoolExecutor over solver subprocesses oversubscribes CPU and memory. Bound concurrency by the actual resource: if each run uses 8 cores and the node has 32, cap at four concurrent jobs. For HPC, let the scheduler handle packing and submit all jobs at once with appropriate --ntasks and --mem requests.

A practical pattern is a bounded worker pool with per-run isolation and a retry policy for transient failures (license checkout timeouts, scratch I/O errors). Retry once with a fresh working directory; if it fails again, mark the design point as failed and continue. Never let one bad run abort the batch.

Putting It Together: A Minimal Batch Driver

The following skeleton ties the pieces into a single reproducible driver. It assumes a parameter list, a template, and a solver command, and it produces a results CSV:

import csv, subprocess
from concurrent.futures import ThreadPoolExecutor, as_completed
from pathlib import Path

def execute(dp, template, solver_cmd, root):
    run_dir = root / dp["id"]
    deck = render_deck(run_dir, dp, template)
    rc, out, err = run_solver(solver_cmd, deck)
    ok = rc == 0 and "COMPLETED SUCCESSFULLY" in (run_dir / f"{deck.stem}.sta").read_text()
    return {"id": dp["id"], "status": "ok" if ok else "failed",
            "objective": parse_objective(run_dir) if ok else None}

with open("results.csv", "w", newline="") as f:
    writer = csv.DictWriter(f, fieldnames=["id", "status", "objective"])
    writer.writeheader()
    with ThreadPoolExecutor(max_workers=4) as pool:
        futures = [pool.submit(execute, dp, tmpl, cmd, root) for dp in design_points]
        for fut in as_completed(futures):
            writer.writerow(fut.result())
            f.flush()

This driver is deliberately small, but it embodies the four properties that make batch CAE automation reliable: deterministic deck generation, isolated execution, explicit success criteria, and incremental result persistence. Scale it by swapping the local executor for a scheduler-backed one; the surrounding logic does not change.