4. Job submission mechanics

Slurm exposes three primary user-facing commands for launching work, and LS-DYNA production runs use all three in different roles. Understanding which command allocates resources, which one launches tasks, and which one merely wraps a script is the foundation for reproducible, scalable explicit-dynamics jobs.

The Three Commands and Their Roles

sbatch submits a batch script to the controller. The controller queues the job, and when resources become available it starts the script on the first node of the allocation. The script then typically calls srun to launch the parallel application. salloc requests an allocation and drops you into a shell on the first node, but it does not run anything by itself — you type srun manually. This distinction matters because a common mistake is to run mpirun or srun outside any allocation, which either fails or silently runs on a single node.

Anatomy of an LS-DYNA Batch Script

A production batch script has four logical regions: Slurm directives, environment setup, working-directory and input handling, and the solver invocation. The following example is a representative template for a double-precision SMP/MPP LS-DYNA run on a cluster with 64 cores per node.

#!/bin/bash
#SBATCH --job-name=dyna_crash
#SBATCH --partition=compute
#SBATCH --nodes=2
#SBATCH --ntasks-per-node=32
#SBATCH --cpus-per-task=1
#SBATCH --time=24:00:00
#SBATCH --mem=0
#SBATCH --output=%x_%j.out
#SBATCH --error=%x_%j.err

module purge
module load intel/2023.2
module load impi/2021.9
module load ls-dyna/13.1.0_mpp_dp

cd $SLURM_SUBMIT_DIR
export LSTC_LICENSE=network
export LSTC_LICENSE_SERVER=27000@license.hpc.local

srun --mpi=pmi2 --cpu-bind=cores \
     ls-dyna i=crash.k ncpu=$SLURM_NTASKS memory=2000m \
     d3plot=50ms endtim=120ms

Several details deserve attention. --mem=0 requests all memory on each node, which is appropriate for memory-hungry crash models but should be used deliberately. --cpus-per-task=1 combined with --ntasks-per-node=32 gives 32 MPI ranks per node, matching the physical core count. The ncpu keyword passed to LS-DYNA must equal $SLURM_NTASKS; a mismatch causes either idle ranks or oversubscription. The --cpu-bind=cores flag pins each rank to a distinct core, which typically improves performance by 5–15% on NUMA systems compared to unbound execution.

Environment Propagation

By default, Slurm exports the submitting shell's environment to the batch job, but this behavior is controlled by --export. The default is --export=ALL, which propagates everything. In production, this is often undesirable because a stale LD_LIBRARY_PATH or a conflicting I_MPI_* variable from an interactive session can silently alter solver behavior. A more robust pattern is to export only what you need:

#SBATCH --export=NONE
module purge
module load intel/2023.2 impi/2021.9 ls-dyna/13.1.0_mpp_dp
export OMP_NUM_THREADS=1
export I_MPI_FABRICS=shm:ofi

With --export=NONE, the job starts with a clean environment, and all required variables are set explicitly in the script. This eliminates an entire class of "works on my login node" failures. Note that SLURM_* variables are always set by Slurm regardless of the export setting, so $SLURM_NTASKS, $SLURM_JOB_ID, and $SLURM_SUBMIT_DIR remain available.

Module Loading Strategy

Module systems (Lmod or Environment Modules) are the standard mechanism for managing compiler, MPI, and solver versions. The critical rule for LS-DYNA is that the MPI library used to build the solver must match the MPI library used to launch it. If ls-dyna/13.1.0_mpp_dp was built against Intel MPI 2021.9, then loading OpenMPI instead will produce runtime errors or, worse, silent data corruption.

A reliable module sequence is:

1. module purge — remove all inherited modules.
2. module load the compiler toolchain (e.g., intel/2023.2).
3. module load the matching MPI (impi/2021.9).
4. module load the LS-DYNA module, which sets LS-DYNA_ROOT and adds the executable to PATH.

Verify the loaded stack before submitting a long job:

module list
which ls-dyna
ldd $(which ls-dyna) | grep -i mpi

The ldd output should show the same MPI library that module list reports. If it shows a different path, the module hierarchy is inconsistent and the job will likely fail at MPI initialization.

Interactive Debugging with salloc

For short tests, salloc provides an interactive allocation that mirrors the batch environment:

salloc --nodes=1 --ntasks-per-node=8 --time=00:30:00 --partition=debug
# once the allocation is granted:
module purge && module load intel/2023.2 impi/2021.9 ls-dyna/13.1.0_mpp_dp
srun --mpi=pmi2 --cpu-bind=cores ls-dyna i=test.k ncpu=8 memory=500m endtim=5ms
exit

This is the fastest way to validate a new input deck, a new module combination, or a changed ncpu setting before committing to a multi-hour production run. The allocation is released when you type exit.

Common Failure Modes and Diagnosis

Three failure patterns account for most LS-DYNA submission problems on Slurm. First, MPI initialization failure — the job starts but srun reports "PMIx" or "unable to contact" errors. This almost always means the MPI library loaded in the script does not match the one the solver was built against. Check with ldd as shown above. Second, ncpu mismatch — LS-DYNA starts but reports fewer or more ranks than expected, or hangs during decomposition. Verify that ncpu=$SLURM_NTASKS and that --ntasks-per-node times --nodes equals the intended rank count. Third, license checkout failure — the solver exits immediately with a license error. Confirm LSTC_LICENSE_SERVER is set and reachable from compute nodes, not just the login node.

Putting It Together: A Reproducible Submission Checklist

Before submitting any production LS-DYNA job, confirm the following: the batch script uses --export=NONE and sets all environment variables explicitly; the module stack is loaded in the correct order (compiler → MPI → solver); ncpu in the LS-DYNA input line equals $SLURM_NTASKS; --cpu-bind=cores is present; the license server variable is set; and the working directory is set to $SLURM_SUBMIT_DIR so that relative input paths resolve correctly. Running through this checklist takes under a minute and prevents the majority of wasted queue time.