4. Configure LS-DYNA MPI execution under Slurm

Why MPI configuration is a separate problem from the batch script

An SMP LS-DYNA job is a single process that spawns threads; the scheduler only has to place one task on one node. An MPP (MPI) job is a set of ranks that must be launched together, discover each other, and be pinned to distinct cores. Slurm allocates the resources, but it does not automatically decide how the ranks are launched or where they land. That decision is made by the launch command (srun or mpirun), the process-binding options, and — for some MPI stacks — a hostfile. Getting these three elements consistent with the allocation is the whole task of this case.

The failure modes when they are inconsistent are distinctive and worth recognizing early: ranks that all land on the first node (oversubscription, near-zero speedup), ranks that cannot find each other (MPI aborts during initialization), or a job that runs but is killed by the cgroup because the launch command asked for more tasks than were allocated.

Reference layout for this case

The example below assumes a two-node allocation with 64 cores per node (128 tasks total), an Intel-MPI-based LS-DYNA MPP executable, and a shared filesystem visible from every node. Substitute your site's actual partition, core count, and executable path; the structure of the configuration does not change.

Building the launch line correctly

The safest pattern on a Slurm cluster is to let srun do the launching. srun inherits the allocation, so the number of tasks, the node list, and the cgroup limits are already known and cannot drift out of sync. A minimal, correct invocation inside the batch script is:

srun --mpi=pmi2 --cpu-bind=cores --distribution=block:block \
     /opt/lsdyna/lsdyna_mpp i=model.key ncpu=128 memory=2000m

Three options carry most of the weight:

• --mpi=pmi2 selects the PMI-2 process-management interface. This must match the MPI library the LS-DYNA MPP binary was built against. If the binary links Intel MPI, pmi2 is normally correct; if it links Open MPI, use --mpi=pmix or --mpi=openmpi depending on the site build. A mismatch typically shows up as an MPI initialization abort before any LS-DYNA banner is printed.
• --cpu-bind=cores pins each rank to a distinct physical core. Without it, the OS scheduler may pack several ranks onto the same core, and the job runs far slower than the core count suggests.
• --distribution=block:block assigns consecutive ranks to consecutive cores on the same node before moving to the next node. This keeps the rank numbering contiguous per node, which matters for LS-DYNA's domain decomposition and for interpreting per-rank timing in the output.

Note that ncpu=128 in the LS-DYNA command line must equal --ntasks in the allocation. LS-DYNA will attempt to start that many ranks; if the allocation provides fewer, the launch fails or the extra ranks are killed by the cgroup.

When mpirun is required instead

Some site builds of LS-DYNA MPP are wrapped so that they expect to be launched by mpirun rather than srun. In that case you must supply a hostfile, because mpirun does not read the Slurm allocation by itself. Generate it from the allocation rather than hard-coding node names:

scontrol show hostnames "$SLURM_JOB_NODELIST" > "$SLURM_SUBMIT_DIR/hostfile"
mpirun -np "$SLURM_NTASKS" -ppn 64 -f hostfile \
       -genv I_MPI_PIN_DOMAIN=core \
       /opt/lsdyna/lsdyna_mpp i=model.key ncpu="$SLURM_NTASKS"

The hostfile produced by scontrol show hostnames contains one hostname per line, which mpirun interprets as one slot per line unless -ppn overrides it. Setting -ppn 64 tells mpirun to place 64 ranks per host, matching the allocation. I_MPI_PIN_DOMAIN=core is the Intel MPI equivalent of --cpu-bind=cores; for Open MPI the equivalent is --bind-to core. Mixing the two families of binding flags is a common source of silent oversubscription.

Verifying the mapping before trusting the run

Before committing a long production job, confirm that the rank-to-core mapping is what you intended. Two cheap checks:

1. Add --cpu-bind=verbose to the srun line for a short test run. Slurm prints a line per task showing the core it was bound to; you should see 64 distinct cores on each node with no duplicates.
2. For Intel MPI, set I_MPI_DEBUG=4 in the environment. The library prints the rank placement table at startup, which lets you confirm that ranks 0–63 are on the first node and 64–127 on the second.

A correct test run shows the LS-DYNA banner once, followed by the MPP initialization message reporting the expected number of processors, and the per-rank timing summary at the end should show roughly balanced CPU times across ranks. If one node's ranks report much higher wall time, the binding or distribution is likely wrong.

Common symptoms and their causes