2. LS-DYNA solver landscape on HPC

Before a single Slurm job script is written, you must decide which LS-DYNA executable to run and how to map it onto the cluster's hardware. LS-DYNA is not one program but a family of binaries that share the same keyword input deck while differing fundamentally in their parallel decomposition strategy. Choosing the wrong one wastes core-hours, triggers license failures, or produces a job that never leaves the queue. This section establishes the solver taxonomy, the licensing mechanics that constrain it, and the version-specific deployment details that determine whether a job actually starts.

The Three Executable Families

LS-DYNA ships three principal parallelization models, each compiled as a distinct binary:

  • SMP (Shared-Memory Parallel) — a single process with multiple OpenMP threads sharing one address space. Historically invoked as ls-dyna_smp or lsdyna_smp. Memory is shared, so the model must fit within one node's RAM. Scaling is typically good to 8–16 threads, then degrades as memory bandwidth saturates.
  • MPP (Massively Parallel Processing) — pure MPI, one process per rank, distributed memory via domain decomposition. Invoked as ls-dyna_mpp or lsdyna_mpp. This is the only model that scales across nodes and is the default choice for large models.
  • Hybrid MPI/OpenMP — MPI ranks each spawning OpenMP threads. Invoked as ls-dyna_hybrid or built with -DOPENMP. Reduces MPI rank count per node, which lowers communication overhead and memory duplication on high-core-count sockets.

The critical practical distinction is memory topology. SMP shares one copy of the model; MPP replicates the mesh across ranks and exchanges boundary data through MPI. A 40-million-element model that fits comfortably in 256 GB on one node under SMP may require careful domain decomposition under MPP to avoid a rank running out of memory.

Choosing the Right Decomposition

The decision is driven by model size, node architecture, and interconnect. The following table summarizes the practical selection criteria:

Model size (elements) Recommended executable Typical layout Rationale
< 1 M SMP 1 node, 4–8 threads MPI overhead exceeds benefit; single-node memory is sufficient
1 M – 10 M SMP or MPP 1 node SMP, or 2–4 nodes MPP Depends on whether the model fits in node RAM
10 M – 100 M MPP 4–32 nodes, 1 rank/core Distributed memory required; InfiniBand essential
> 100 M Hybrid MPI/OpenMP Many nodes, 2–4 threads/rank Reduces rank count and MPI message volume

On modern nodes with 64–128 cores, pure MPP with one rank per core generates enormous MPI message counts. Hybrid mode with, say, 16 ranks × 4 threads per node often outperforms 64 ranks × 1 thread because it cuts inter-rank communication by a factor of four while keeping all cores busy. The trade-off is that OpenMP thread scaling within a rank is sublinear, so the sweet spot must be measured, not assumed.

Rule of thumb: Start with MPP at one rank per physical core. If profiling shows MPI communication dominating (look for high MPI_Wait or MPI_Allreduce time in the LS-DYNA timing summary), switch to hybrid with 2–4 threads per rank and re-measure. Never exceed the number of physical cores per node with the product of ranks × threads.

Licensing Models and Their Scheduling Impact

LS-DYNA licensing directly shapes how Slurm must be configured. Three models are common in production:

  • Node-locked / per-seat — a fixed license tied to a host or user. Only one job per license. Slurm must enforce a license-count resource so jobs queue rather than fail at startup.
  • Floating (network) licenses — served by LSTC's license daemon (lstc_server) or a FlexLM-style manager. The number of concurrent jobs is capped by available tokens, independent of core count.
  • Core-count / token-based — the license grants a pool of cores (e.g., 512 cores total). A 64-core job consumes 64 tokens. This is the most common HPC arrangement and the one most likely to cause "license denied" failures when jobs oversubscribe.

Under a core-count model, the Slurm --ntasks and --cpus-per-task values must multiply to a number the license server can grant. If the pool is exhausted, LS-DYNA exits immediately with a message such as License error: cannot connect to license server or Not enough licenses available. The correct response is not to retry blindly but to configure a Slurm license resource:

# In slurm.conf, declare the license
Licenses=lsdyna:512

# In the job script, request it
#SBATCH --licenses=lsdyna:64
#SBATCH --ntasks=64
#SBATCH --cpus-per-task=1

With this configuration, Slurm holds jobs in PENDING with reason Licenses until tokens free up, rather than letting them start and crash. This is far more efficient than relying on the solver's own license check.

Common failure: A user requests --ntasks=128 but the license pool only has 64 cores free. Without a Slurm license resource, the job starts, LS-DYNA launches, and the solver aborts after allocating memory. The node is occupied for minutes before failing. Always declare the license in slurm.conf and request it in the job script.

Version-Specific Deployment Considerations

LS-DYNA versions differ in compiler requirements, MPI library compatibility, and keyword support. Deployment pitfalls cluster around three areas:

  1. MPI library matching. An MPP binary built against OpenMPI 4.x will not run correctly under MVAPICH2 or Intel MPI. The mpirun or srun launcher must match the build. Check with ldd ls-dyna_mpp | grep mpi before deploying. On Slurm, prefer srun over mpirun when the binary is built against the system MPI, because srun integrates with Slurm's task placement and cgroup accounting.
  2. Compiler runtime libraries. Intel-compiled binaries need libiomp5 and libifcore in LD_LIBRARY_PATH. GCC-compiled binaries need libgomp and libgfortran. A missing runtime produces error while loading shared libraries at launch. Module systems (module load ls-dyna/13.1) should set these automatically; verify with module show.
  3. Keyword and feature drift. Newer versions add cards (e.g., *DEFINE_CPM_..., *MAT_... revisions) that older binaries silently ignore or reject. A deck that runs on R11 may fail on R13 with unknown keyword. Pin the version in the job script and document it in the model's README.

The following diagram shows how the solver choice, license pool, and Slurm resource request interact at job submission time:

LS-DYNA Solver Selection and License Gating in Slurm Input Deck keyword file + mesh element count, memory Model size & node RAM? SMP 1 node, OpenMP fits in RAM MPP multi-node, MPI distributed Hybrid MPI/OpenMP ranks × threads high core count License Pool lstc_server / FlexLM 512 core tokens concurrent job cap Slurm Controller Licenses=lsdyna:512 --licenses=lsdyna:N holds job if tokens unavailable Job Outcome RUNNING → solver starts PENDING (Licenses) FAILED (no tokens) Troubleshooting check ldd, module load, license log Legend: SMP MPP Hybrid License Outcome

Deployment Checklist for a New Cluster Node

When onboarding a new node type or upgrading LS-DYNA, verify the following before submitting production jobs:

  • Confirm the binary's MPI vendor: ldd $LSDYNA_ROOT/ls-dyna_mpp | grep -i mpi
  • Confirm OpenMP runtime presence: ldd $LSDYNA_ROOT/ls-dyna_smp | grep -i omp
  • Verify the license server is reachable: telnet license-host 31011 (or the configured port)
  • Check that slurm.conf declares the license and that scontrol show lic reports the correct total
  • Run a small benchmark deck (e.g., the LS-DYNA example benchmark.k) at 1, 8, 32, and 64 cores to establish scaling baselines
  • Record the version string from ls-dyna_mpp -v and pin it in the module file

These checks take minutes and prevent the most common class of production failures: jobs that start, consume node time, and abort due to a missing library or exhausted license pool. With the solver family, license model, and version constraints understood, the next step is translating them into a correct Slurm batch script.

← Previous 📖 Contents Next →