4. Job submission mechanics

A crash CAE job on a shared cluster is not a single executable invocation. It is a bundle of solver-specific inputs, restart state, license entitlements, runtime environment, and scheduler directives that must all agree before the first time step is computed. When LS-DYNA, PAM-CRASH, Radioss, or Abaqus Explicit jobs share the same queue, the submission layer becomes the place where most avoidable failures originate. This section walks through the mechanics of constructing a correct submission for a multi-solver crash workload.

Input decks and solver-specific invocation

Each explicit crash solver consumes a different deck format and expects a different command line. LS-DYNA reads a keyword deck (main.k) plus optional include files; PAM-CRASH uses a .pc input with a separate .erf execution control file; Radioss uses a starter/engine split (Starter then Engine); Abaqus Explicit uses an .inp deck driven by abaqus job=.... The scheduler script must therefore be solver-aware rather than generic.

A minimal LS-DYNA submission inside a job script looks like this:

#SBATCH --job-name=crash_lsdyna
#SBATCH --nodes=2
#SBATCH --ntasks-per-node=32
#SBATCH --time=08:00:00
#SBATCH --partition=explicit

module load lsdyna/R13.1
export LSTC_LICENSE=network
export LSTC_LICENSE_SERVER=27000@lic-srv

srun -n 64 lsdyna i=main.k memory=800m ncpu=64

The i= flag names the deck, memory= sets the per-process memory pool, and ncpu= must match the MPI rank count. A mismatch between ncpu and srun -n is one of the most common silent failures: the solver starts, decomposes the model incorrectly, and either hangs or produces non-physical results.

Restart files and continuation logic

Crash simulations are long and frequently interrupted by wall-clock limits. Restart handling is not optional. LS-DYNA writes d3dump01, d3dump02, and so on at intervals controlled by *DATABASE_BINARY_D3DUMP and *CONTROL_TERMINATION. To continue, the job script must point to the latest dump:

lsdyna i=main.k r=d3dump03 memory=800m ncpu=64

For PAM-CRASH the restart is triggered by the -restart flag referencing the .rst file, and Radioss uses Engine with a -restart argument pointing to the run directory. The critical operational rule is that the restart file must be on a filesystem visible to every compute node — a local /tmp dump is useless after a node failure. Always direct restart output to the shared parallel filesystem via *DATABASE_BINARY_D3DUMP path settings or the solver's equivalent.

License tokens and concurrent solver access

Crash solvers are license-metered, and multi-solver clusters often share a single license server pool. LS-DYNA typically consumes one token per MPI rank or per node depending on the license model; PAM-CRASH and Radioss have their own token schemes. If the scheduler launches 64 ranks and only 40 tokens are free, the job will either fail immediately or, worse, start and stall waiting for tokens.

Two practical controls help. First, set the license environment variables explicitly in the job script so the solver contacts the correct server:

export LSTC_LICENSE_SERVER=27000@lic-srv
export PAM_LICENSE_FILE=27000@lic-srv
export RADIOSS_LICENSE_SERVER=27000@lic-srv

Second, use scheduler resource limits to cap concurrent token consumption. Slurm's --licenses=lsdyna:64 directive, when the license is registered with the scheduler, prevents oversubscription by holding the job in a pending state until tokens are available. This is far safer than letting the solver fail mid-decomposition.

Environment modules and runtime consistency

Shared clusters almost always use Environment Modules or Lmod. The module set determines the MPI implementation, compiler runtime, and solver build. A job that loads openmpi/4.1.5 while the solver was built against intel-mpi/2021 will fail at MPI_Init with an ABI mismatch. The job script must load modules in the same order and version that the solver was validated against:

module purge
module load intel/2023.2
module load intel-mpi/2021.10
module load lsdyna/R13.1

Record the module list in the job's output for reproducibility. When a run succeeds, the exact module stack is part of the result provenance.

Multi-solver job scripts and resource partitioning

Some crash workflows run multiple solvers in sequence — for example, a Radioss occupant model coupled with an LS-DYNA structural model, or a PAM-CRASH run followed by a post-processing step. These should be expressed as a single job script with explicit stages, or as a job array with dependencies. The key is that each stage declares its own module stack, license variables, and rank count, because the solvers do not share a runtime.

Verification before committing cluster time

Before submitting a multi-hour crash run, perform a short smoke test: reduce *CONTROL_TERMINATION to a few milliseconds of simulated time, submit with the same module stack and rank count, and confirm that the solver initializes, reads the restart file if applicable, and writes a dump. Check the d3hsp or equivalent header file for the reported rank count, memory allocation, and license checkout. If the smoke test passes, the full run is likely to start cleanly. If it fails, the error is almost always in one of the four layers shown above — deck, restart, license, or module — and can be isolated by changing one variable at a time.

The submission layer is where reproducibility is won or lost. A job script that records its module stack, license server, rank count, restart source, and dump interval is not just a convenience — it is the minimum evidence needed to reproduce a crash result months later on the same shared cluster.