4. Building reproducible LS-DYNA environments

An LS-DYNA environment is reproducible only when three independent layers are locked together: the solver binary itself, the license client stack, and the MPI runtime that the binary was linked against. A common failure mode in crash CAE teams is that a job runs correctly on one node and fails, hangs, or produces divergent results on another — not because the model changed, but because one of these three layers drifted. This section treats each layer as a versioned artifact and shows how to pin them so that a given input deck always maps to a known, documented execution environment.

Why the Solver Binary Is Not a Single File

LS-DYNA ships as several distinct executables rather than one universal binary. The most relevant variants for crash work are:

• SMP (ls-dyna_smp_*) — shared-memory only, no MPI dependency, used for single-node runs and for debugging models before scaling out.
• MPP (ls-dyna_mpp_*) — distributed-memory, linked against a specific MPI implementation (Intel MPI, Open MPI, MPICH, or platform MPI).
• Hybrid — MPP binary launched with multiple threads per rank, combining MPI ranks with OpenMP threads.

Each binary is compiled for a specific precision (single or double), a specific compiler family, and a specific MPI ABI. The filename usually encodes this, for example ls-dyna_mpp_d R11.1.0_intelmpi_avx2_linux_x64. The critical rule is that the MPI library the binary was linked against must match the MPI library present at runtime. Mixing an Intel MPI–linked binary with an Open MPI mpirun typically produces an immediate abort or, worse, a silent hang during domain decomposition.

Pinning the MPI Runtime

MPI pinning means ensuring that the mpirun/mpiexec launcher, the MPI shared libraries, and the fabric libraries (InfiniBand verbs, libfabric) all come from one consistent installation. On a cluster with a module system, this is done by loading a single MPI module and never mixing it with another.

# Example: pin Intel MPI 2021.9 for an LS-DYNA MPP run
module purge
module load intel-mpi/2021.9
module load ls-dyna/11.1.0-intelmpi

# Verify the launcher and libraries are consistent
which mpirun
mpirun --version
ldd $(which ls-dyna_mpp_d) | grep -i mpi

The ldd output should reference libmpi.so from the same Intel MPI installation that mpirun belongs to. If it references a system /usr/lib64/libmpi.so instead, the environment is not pinned and results are not reproducible.

For fabric-level reproducibility, also pin the interconnect provider. On InfiniBand clusters, set I_MPI_FABRICS=shm:ofi or shm:dapl explicitly rather than relying on auto-detection, because auto-detection can select different transports on nodes with different firmware or driver versions.

License Integration

LS-DYNA uses FlexNet (FlexLM) licensing. The solver reads the license server location from the LSTC_LICENSE environment variable or from a license file. Two settings matter for reproducibility:

• LSTC_LICENSE=network with LSTC_LICENSE_SERVER=port@host — the standard cluster configuration.
• LSTC_LICENSE=ansys — used when the license is served through an Ansys license manager rather than LSTC directly.

License checkout behavior affects reproducibility in a subtle way: if a job requests more MPI ranks than the license allows, LS-DYNA may fall back to fewer ranks or abort mid-decomposition. Pin the license server and the expected license feature in the environment manifest, and verify checkout before launching a long run:

export LSTC_LICENSE=network
export LSTC_LICENSE_SERVER=29000@licenses.example.org

# Confirm the license feature is reachable before submitting
lmutil lmstat -c 29000@licenses.example.org -f MPPDYNA

If lmstat reports the feature as unavailable, the job will fail at startup with a license error rather than a solver error — a distinction that matters when triaging failures.

Putting It Together: A Reproducible Launch Recipe

The following recipe pins all three layers for a 64-rank MPP run on two nodes with 32 cores each. It is written as a job script fragment and is intended to be the single entry point for this environment.

#!/bin/bash
#SBATCH --job-name=dyna-repro
#SBATCH --nodes=2
#SBATCH --ntasks-per-node=32
#SBATCH --cpus-per-task=1

module purge
module load intel-mpi/2021.9
module load ls-dyna/11.1.0-intelmpi

export LSTC_LICENSE=network
export LSTC_LICENSE_SERVER=29000@licenses.example.org
export I_MPI_FABRICS=shm:ofi
export I_MPI_PIN=1

# Record the environment for the run manifest
echo "binary: $(which ls-dyna_mpp_d)"
sha256sum $(which ls-dyna_mpp_d)
mpirun --version | head -n 1

mpirun -np 64 ls-dyna_mpp_d i=crash_v3.k ncpu=64 memory=2000m

The ncpu value must match the number of MPI ranks; a mismatch causes LS-DYNA to either under-utilize the allocation or abort during decomposition. The memory value is per-rank in megabytes and must be sized so that the total across ranks fits within node RAM.

Verifying Reproducibility

After a run completes, the observable verification criteria are:

• The d3hsp file reports the expected number of CPUs and the correct binary version string.
• The messag file shows no MPI warnings about mismatched libraries or fabric fallback.
• The glstat and matsum files show energy conservation within the expected tolerance for the model.
• Re-running the same deck on a different node with the same pinned environment produces bitwise-identical or statistically equivalent results, depending on whether the model uses floating-point reduction order–sensitive operations.

Typical failure symptoms and their causes: an immediate abort with MPI_Init errors indicates an MPI ABI mismatch; a hang during domain decomposition usually indicates a fabric or launcher mismatch; a license error at startup indicates the license server or feature is unreachable; and divergent results across nodes with identical binaries usually indicate a floating-point reduction order difference introduced by a different MPI rank count or a different thread count.

Environment Manifest as a Deliverable

The practical output of this section is a manifest that travels with every result set. It should be short enough to paste into a job log and complete enough to reconstruct the environment. A minimal manifest for the example above:

When a result is questioned weeks later, this manifest is what allows the team to distinguish a modeling change from an environment change. Without it, the only recourse is to re-run and hope the environment has not drifted — which is precisely the situation reproducible HPC environments are meant to eliminate.