4. Simulation data capture and ingestion

By the time a CAE job finishes, the most valuable artifact is rarely the single result file a solver writes at the end. It is the context around that result: which input deck was used, which solver build and license feature were consumed, which environment variables were set, how many cores and how much memory the job actually used, whether it converged, and what the scheduler recorded about exit status and wall-clock time. Manual collection of this context is where simulation data management most often fails — not because engineers are careless, but because the data lives in three different places (solver output, wrapper scripts, and the job scheduler) and none of them knows about the others. This chapter describes how to harvest all three automatically and land them in a structured, queryable store.

What "capture" actually means in a CAE pipeline

Capture is the process of extracting metadata and result artifacts from a completed (or failed) simulation run and normalizing them into a common schema before ingestion. It is distinct from ingestion, which is the act of writing that normalized record into the repository (database, object store, or PLM system) together with the associated files. A robust capture layer answers four questions for every run:

• Identity — what was run: model revision, input deck hash, solver name and version, case/variant identifier.
• Provenance — how it was run: command line, wrapper script version, environment modules, scheduler job ID, queue, resource request.
• Outcome — did it succeed: solver exit code, convergence status, error/warning counts, wall-clock and CPU time, peak memory.
• Artifacts — what it produced: result files, log files, images, and their checksums and sizes.

The key design principle is that capture must be passive and non-intrusive: it should never require the analyst to remember to run a separate upload step, and it should never modify the solver's own output. Everything is derived from files and scheduler records that already exist.

Harvesting from the solver

Solvers expose outcome information in predictable places. For a typical implicit structural solver, the relevant signals are the exit code, the tail of the .out or .log file, and the convergence history. A capture script should parse these deterministically rather than relying on a human reading them. For example, a small Python harvester that runs as the last line of the job script can extract the exit status and scan the log for known patterns:

import re, json, hashlib, pathlib, sys

log = pathlib.Path(sys.argv[1])
text = log.read_text(errors="replace")

record = {
    "solver_exit_code": int(sys.argv[2]),
    "log_sha256": hashlib.sha256(log.read_bytes()).hexdigest(),
    "converged": bool(re.search(r"ANALYSIS COMPLETE", text)),
    "warnings": len(re.findall(r"^\s*WARNING", text, re.M)),
    "errors":   len(re.findall(r"^\s*ERROR", text, re.M)),
    "iterations": len(re.findall(r"^\s*ITERATION", text, re.M)),
}
print(json.dumps(record))

Two cautions apply. First, pattern matching against log text is brittle across solver versions; pin the patterns to the solver version recorded in the same run, and treat an unmatched pattern as unknown rather than failed. Second, the exit code alone is insufficient — many solvers return 0 even when the analysis diverged but wrote a partial result. Convergence must be judged from the solver's own completion marker, not from the process status.

Harvesting from wrapper scripts

The wrapper script is the natural place to record provenance, because it is the only component that knows the full command line and the environment. A disciplined wrapper writes a small JSON sidecar before invoking the solver and completes it afterward:

#!/bin/bash
# run_case.sh — invoked by the scheduler
set -euo pipefail
CASE_DIR="$1"; shift
SIDECAR="$CASE_DIR/run_meta.json"

cat > "$SIDECAR" <<EOF
{"case_dir": "$CASE_DIR",
 "cmdline": "$*",
 "host": "$(hostname)",
 "modules": "$(module list 2>&1 | tr '\n' ';')",
 "start_utc": "$(date -u +%FT%TZ)"}
EOF

set +e
"$@" > "$CASE_DIR/solver.log" 2>&1
RC=$?
set -e

python3 harvest.py "$CASE_DIR/solver.log" "$RC" >> "$SIDECAR"
exit $RC

This pattern has one important property: the sidecar is written before the solver starts, so even a job that is killed by the scheduler leaves a provenance record behind. The harvester appends outcome fields afterward. If the job dies mid-run, the sidecar exists but lacks outcome fields — which is itself a useful signal that the run was interrupted rather than completed.

Harvesting from the HPC job scheduler

The scheduler holds the resource and timing truth that neither the solver nor the wrapper can see: queue wait time, actual wall-clock, allocated versus used cores, peak memory, and the scheduler's own exit disposition. For Slurm, sacct provides this in machine-readable form:

sacct -j "$SLURM_JOB_ID" --parsable2 \
  --format=JobID,JobName,State,ExitCode,Elapsed,AllocCPUS,MaxRSS,NodeList

For PBS Pro the equivalent is qstat -xf $PBS_JOBID parsed for resources_used and exit_status. The capture layer should run this query after the job reaches a terminal state, typically from an epilog script or a short-lived post-job collector, because MaxRSS and Elapsed are only final once the job has ended. A common mistake is to query sacct from inside the running job — the values are then incomplete or zero.

End-to-end capture flow

The three sources converge into a single normalized record. The diagram below shows the flow from job submission through the three harvest points to the ingestion store, including the failure branch where a job is killed before the solver completes.

Normalization and ingestion

Each source produces a partial record; the normalizer merges them into one schema keyed by a stable run identifier. A practical key is a hash of (model_revision, input_deck_sha256, solver_version, cmdline), which makes duplicate submissions of the identical case collapse to the same key — useful for detecting accidental re-runs. The merged record should carry explicit nulls for missing fields rather than omitting them, so that downstream quality checks can distinguish "not captured" from "captured as zero."

Ingestion then writes the record and uploads artifacts. Two rules keep this reliable: upload artifacts first and write the database record last, so a record never points at a missing file; and compute checksums at capture time, not at ingestion time, so the checksum reflects the file as the solver wrote it.

Failure symptoms and troubleshooting

The result of a correctly built capture layer is that every run — successful, diverged, or killed — leaves a complete, queryable trace in the repository without any analyst action. That trace is the foundation for the traceability and quality gates developed in later chapters: you cannot audit convergence, compare variants, or reproduce a result if the run's provenance and outcome were never captured in the first place.