3. Job submission and scheduling for LS-DYNA and Pam-Crash

Crash CAE workloads are long-running, memory-hungry, and highly parallel, which makes them a poor fit for interactive execution on login nodes. In production HPC environments, every LS-DYNA or Pam-Crash run should be submitted through a batch scheduler — typically Slurm, PBS Pro, or LSF — so that the cluster can arbitrate CPU, memory, and license resources fairly across teams. This section focuses on the practical mechanics: how to write a batch script, how to translate a solver's parallelism model into a resource request, and how queue policies shape what you can actually get.

From Solver Parallelism to Resource Request

Both LS-DYNA and Pam-Crash support two distinct parallelism modes, and the resource request must match the mode you intend to run:

• Shared-memory (SMP) — a single process with multiple threads, controlled by NCPU (LS-DYNA) or -nt (Pam-Crash). The scheduler sees one task; you request one node with many cores.
• Distributed-memory (MPP/MPI) — multiple ranks communicating over MPI, controlled by NCPU plus a decomposition file (decomp for LS-DYNA, pc-decomp for Pam-Crash). The scheduler sees N tasks; you request N tasks across one or more nodes.

A common production mistake is requesting 64 cores but launching only 8 MPI ranks, or vice versa. The scheduler allocates what you asked for, the solver uses what you told it, and the mismatch either wastes allocation or triggers oversubscription. The rule is simple: the number of MPI ranks must equal the number of tasks requested, and for SMP the thread count must equal the cores on the allocated node.

Anatomy of a Production Batch Script

The following Slurm script is a realistic template for an LS-DYNA MPP run. It is illustrative — adapt paths, partitions, and license settings to your site.

#!/bin/bash
#SBATCH --job-name=crash_frontal_01
#SBATCH --partition=crash
#SBATCH --nodes=2
#SBATCH --ntasks=64
#SBATCH --ntasks-per-node=32
#SBATCH --cpus-per-task=1
#SBATCH --mem=180G
#SBATCH --time=24:00:00
#SBATCH --output=slurm-%j.out
#SBATCH --error=slurm-%j.err

module purge
module load ls-dyna/13.1.0
module load intel-mpi/2021.9

export LSTC_LICENSE=network
export LSTC_LICENSE_SERVER=27000@lic-srv.corp

cd $SLURM_SUBMIT_DIR

srun -n 64 --mpi=pmi2 \
  ls-dyna i=frontal_01.k \
           ncpu=64 \
           memory=2000m \
           d=decomp_frontal_01 \
           jobid=frontal_01 \
           endtim=0.12

Several details deserve attention. --ntasks=64 with --ntasks-per-node=32 pins 32 ranks per node, which matches the physical core count and avoids hyperthread contention. --mem=180G is a per-node request in Slurm; for a crash model with roughly 8–12 million solid elements, 180 GB per node is a reasonable starting point, but you should verify against the solver's memory estimate printed at startup. The memory=2000m keyword inside LS-DYNA is the per-rank memory pool, not the total — do not confuse the two.

For Pam-Crash, the equivalent launch line changes:

srun -n 64 --mpi=pmi2 \
  pamcrash -nt 64 \
           -np 64 \
           -mem 180000 \
           -dir $SLURM_SUBMIT_DIR \
           frontal_01.pc

Pam-Crash uses -nt for threads and -np for MPI processes; on a pure MPP run they are typically equal. The -mem flag is in megabytes and applies to the whole job, unlike LS-DYNA's per-rank memory=.

Queue Policies That Determine What You Get

Batch scripts do not run in a vacuum. Queue policies — partition limits, fair-share, backfill, and license-aware scheduling — decide whether your job starts in five minutes or five hours. The most consequential policies for crash workloads are:

Two practical consequences follow. First, always set --time to a realistic value. Overestimating blocks backfill; underestimating kills the job. A good heuristic is to run a 1-hour pilot with the same decomposition, measure the timestep rate, and extrapolate to the full termination time with a 20% margin. Second, check license availability before submitting a large MPP job. A 64-rank LS-DYNA run may consume 64 tokens; if only 40 are free, the job will sit in a license queue regardless of CPU availability.

Choosing the Right Decomposition

MPP scaling for crash models is not linear. Contact-heavy models — full vehicle frontal impact, side pole, or occupant restraint simulations — often scale poorly beyond 32–64 ranks because contact search and communication dominate. A useful diagnostic is to compare the "elapsed time per cycle" reported in the d3hsp file across two runs with different rank counts. If doubling ranks reduces cycle time by less than 30%, you are past the efficient scaling region and should consider SMP or a hybrid approach instead.

Monitoring and Failure Symptoms

Once submitted, the job's health can be judged from three sources: the scheduler's accounting (sacct -j <jobid>), the solver's own log (d3hsp for LS-DYNA, .out for Pam-Crash), and the node-level metrics if available. Common failure patterns and their causes:

• Job exits immediately with "out of memory" — the per-rank memory= or -mem is too low, or --mem per node is insufficient. Increase both, and check the solver's memory estimate in the first 20 lines of the log.
• Job runs but stalls at "initializing MPI" — rank count mismatch between srun -n and the solver's ncpu=/-np. Align them.
• Job killed at walltime — the endtim is too large for the allocated time. Restart from the last d3dump file with a longer --time.
• Job pending with reason "Licenses" — solver tokens exhausted. Wait, or reduce rank count to consume fewer tokens.

Putting It Together: A Submission Checklist

Before hitting sbatch, verify the following. This checklist is deliberately short — it captures the items that most often cause wasted allocation or failed runs:

1. Rank count in srun -n equals ncpu= (LS-DYNA) or -np (Pam-Crash).
2. --ntasks-per-node does not exceed physical cores per node.
3. --mem per node is at least 1.5× the solver's estimated peak memory.
4. --time is based on a measured pilot run, not a guess.
5. License server environment variables are exported before the solver launch.
6. The decomposition file exists and its rank count matches the launch.
7. Restart files (d3dump or equivalent) are written at a frequency that survives a walltime kill.

With these practices in place, LS-DYNA and Pam-Crash jobs move through the queue predictably, use the allocated resources efficiently, and produce diagnostic output that makes failures quick to isolate rather than mysterious.