4. Job submission automation on HPC cluster

On a shared HPC cluster, LS-DYNA is almost never launched by typing lsdyna at a login prompt. The scheduler owns the compute nodes, and the only legitimate way to reach them is through a batch script that declares resources, sets the environment, and invokes the solver. This chapter focuses on the mechanics of that script across the three dominant workload managers — SLURM, PBS/Torque, and LSF — and on the LS-DYNA-specific details that determine whether a job runs correctly once it starts.

What the scheduler actually needs from you

All three schedulers answer the same three questions before a job can start: how many resources, for how long, and what command to run. The directive syntax differs, but the semantics are nearly identical. The table below maps the common controls; treat it as a translation layer when moving a script between clusters.

Two conventions matter more than the syntax. First, always request whole nodes for LS-DYNA MPP runs: the solver's domain decomposition assumes it can use every core it is given, and partial-node allocations invite oversubscription when the scheduler packs other jobs onto the same node. Second, set wall time deliberately. LS-DYNA writes restart dumps at intervals you control, so a job that hits the wall-time limit is not lost — but only if you planned for it.

Invoking the LS-DYNA solver correctly

The executable name varies by installation (lsdyna, ls-dyna_mpp_d, mpp-dyna), and the launch mechanism depends on whether the build is SMP or MPP. For MPP builds, the solver must be started through the scheduler's parallel launcher so that ranks are placed on the allocated nodes. The pattern is consistent:

# SLURM
srun --mpi=pmix_v3 lsdyna_mpp_d i=crash.k ncpu=$SLURM_NTASKS memory=2000m

# PBS / Torque
mpirun -np $PBS_NP lsdyna_mpp_d i=crash.k ncpu=$PBS_NP memory=2000m

# LSF
mpirun.lsf lsdyna_mpp_d i=crash.k ncpu=$LSB_DJOB_NUMPROC memory=2000m

The ncpu keyword must match the number of ranks the launcher actually starts. A mismatch is the single most common cause of a job that starts, prints a banner, and then hangs or aborts with a decomposition error. The memory= keyword is per-rank in most builds; setting it too low triggers a clean "insufficient memory" exit, while setting it too high can cause the scheduler to reject the job at submission time if memory is a tracked resource.

Anatomy of a production batch script

The following SLURM script is a complete, runnable template. It stages the input, runs the solver, and preserves restart files so a wall-time kill does not destroy the run.

#!/bin/bash
#SBATCH -J crash_v3
#SBATCH -p normal
#SBATCH -N 2
#SBATCH --ntasks-per-node=32
#SBATCH -t 24:00:00
#SBATCH -o %x-%j.out
#SBATCH -e %x-%j.err

set -euo pipefail
module purge
module load intel/2023.2 openmpi/4.1.5 lsdyna/13.1

cd "$SLURM_SUBMIT_DIR"
cp crash.k "$SLURM_JOB_ID".k

srun --mpi=pmix_v3 lsdyna_mpp_d \
     i="$SLURM_JOB_ID".k \
     ncpu="$SLURM_NTASKS" \
     memory=2000m \
     d3dump=1 \
     d3dump_interval=3600

# Preserve restart and message files for post-processing
mkdir -p "results/$SLURM_JOB_ID"
mv d3dump* messag* d3hsp "results/$SLURM_JOB_ID/" 2>/dev/null || true

Three details in this script are worth calling out. The set -euo pipefail line ensures the script aborts on any unset variable or failed command, which prevents a silent partial run. Copying the input to a job-ID-named file avoids collisions when several jobs share a directory. And d3dump_interval=3600 writes a restart dump every simulated hour, so a job killed at the 24-hour wall limit can be resumed with r= rather than restarted from zero.

Dependency chains and array jobs

Real studies rarely consist of one run. A parameter sweep of twenty variants should be submitted as a job array, not twenty separate sbatch calls. In SLURM, #SBATCH --array=1-20 exposes $SLURM_ARRAY_TASK_ID, which you use to select the input file. PBS uses #PBS -t 1-20 with $PBS_ARRAYID; LSF uses #BSUB -J "sweep[1-20]" with $LSB_JOBINDEX. For sequential stages — mesh generation, then solve, then post-processing — submit with a dependency so the second job waits for the first to succeed:

# SLURM: run post-processing only after the solve completes successfully
sbatch --dependency=afterok:$SOLVE_JOBID post.sh

# PBS: equivalent dependency
qsub -W depend=afterok:$SOLVE_JOBID post.pbs

# LSF: equivalent dependency
bsub -w "done($SOLVE_JOBID)" < post.lsf

Using afterok rather than afterany is important: if the solve fails, you do not want post-processing to run against incomplete results and produce misleading plots.

Diagnosing a job that will not start or will not finish

Most submission failures fall into four categories, and each has a distinct signature. A job stuck in PENDING with reason Resources means the request exceeds what any node can satisfy — check that --ntasks-per-node does not exceed the physical core count. A job that starts and exits within seconds usually indicates a module or MPI mismatch; inspect the .err file for PMI/PMIx messages. A job that runs but produces no d3hsp output is typically a working-directory problem — the solver ran somewhere other than where you expected. And a job killed at wall time without a restart dump means d3dump_interval was never set or was set longer than the wall time.

The diagram above traces the full lifecycle: the batch script declares resources and loads the solver module, the scheduler allocates nodes and launches ranks, LS-DYNA writes its output set, and the restart path feeds a new submission when a run is interrupted. The failure panel at the bottom maps the four most common symptoms to their first diagnostic check, which is where most debugging time is actually spent.

Portability checklist

When moving a working script between clusters, verify these items in order: the module names and versions for the solver and MPI; the launcher command and its MPI flag (--mpi=pmix_v3 on SLURM is not universal); the per-rank memory keyword semantics; the queue or partition name; and the array-index variable name. A script that runs on one cluster and fails on another almost always differs on one of these five points, not on the LS-DYNA input deck itself.