4. Cluster provisioning and node imaging

Crash simulation clusters are not general-purpose HPC clusters. A node that runs LS-DYNA, PAM-CRASH, or Radioss must satisfy a tightly coupled set of requirements: a pinned kernel and glibc, a specific NVIDIA driver branch matched to the CUDA runtime, an MPI implementation that the solver binary was actually linked against, and a filesystem client that behaves correctly under millions of small writes. Provisioning is therefore the discipline of freezing all of these into a reproducible image, then deploying that image identically across every compute node. The goal is not "a working node" but "a node that is bit-identical to the validated reference node."

Why Imaging Beats Per-Node Configuration

In a 200-node crash cluster, a single divergent library version can produce non-reproducible results or, worse, silent numerical differences that only appear after a 14-hour explicit run. Configuration drift is the dominant source of "it worked yesterday" incidents. Node imaging solves this by treating the entire OS plus solver stack as one immutable artifact. Two approaches dominate:

• Golden image cloning — build one reference node, capture it (e.g., with dd, Clonezilla, or a vendor tool), and multicast it to all nodes. Fast to deploy, but the image is monolithic and hard to patch selectively.
• Image + configuration management — a base image provides the OS and drivers, while Ansible, Puppet, or a post-boot script injects node-specific identity (hostname, IP, MPI rank files). This is the more maintainable pattern for clusters that change over time.

Most production crash clusters use the second model: a base image for the heavy, rarely-changing layers (kernel, driver, CUDA, MPI) and configuration management for the light, frequently-changing layers (scheduler config, mount points, user accounts).

The Layered Dependency Stack

The critical insight is that crash solver dependencies form a strict bottom-up stack. Each layer constrains the one above it, and a mismatch anywhere propagates upward as a runtime failure or a numerical inconsistency.

Building the OS Baseline

Start from a vendor-supported enterprise Linux — Rocky Linux 8.x or RHEL 8.x are common in crash shops because solver vendors certify against them. The baseline must pin the following:

• Kernel version — crash solvers are sensitive to kernel scheduler and memory-management behavior. Pin to a specific minor release (e.g., 4.18.0-513.el8) and disable automatic kernel updates via dnf versionlock.
• glibc and libstdc++ — these are the most common source of "works on node A, fails on node B." Record the exact versions with rpm -q glibc libstdc++ and lock them.
• OFED / InfiniBand stack — the userspace verbs library must match the kernel module and the HCA firmware. A mismatch here produces intermittent MPI hangs that are extremely hard to diagnose.
• Filesystem clients — Lustre or GPFS client versions must match the server side. Crash runs generate heavy small-file I/O during contact initialization.

A practical baseline build script captures the reference state:

# Capture the validated baseline manifest
rpm -qa --qf '%{NAME}-%{VERSION}-%{RELEASE}.%{ARCH}\n' | sort > /opt/baseline/rpm-manifest.txt
uname -r > /opt/baseline/kernel.txt
nvidia-smi --query-gpu=driver_version --format=csv,noheader > /opt/baseline/gpu-driver.txt
mpirun --version 2>&1 | head -1 > /opt/baseline/mpi-version.txt
ldd /opt/lsdyna/lsdyna_mpp_d R2024 | grep -E 'mpi|cuda|mkl' > /opt/baseline/solver-libs.txt

These five files become the acceptance criteria for every node. If a node's manifest differs, it is not admitted to the production partition.

Driver and CUDA Alignment

GPU-accelerated crash solvers (LS-DYNA R12+ with GPU offload, or Radioss GPU) require the NVIDIA driver branch to match the CUDA runtime the solver was compiled against. The relationship is not one-to-one: a driver supports a range of CUDA versions, but the solver binary expects a specific one. The safe rule is to install the driver that the solver vendor lists in their platform support matrix, then verify with:

nvidia-smi          # shows driver version and max supported CUDA
nvcc --version      # shows the CUDA toolkit version
ldd /opt/lsdyna/lsdyna_mpp_d | grep libcudart  # shows what the solver links

If nvidia-smi reports CUDA 12.2 but the solver links libcudart.so.11.0, you have a mismatch. The fix is either to install the matching CUDA compatibility package or to rebuild the solver — the former is almost always preferred in production.

MPI Stack Selection and Binding

Crash solvers are typically distributed as MPI-linked binaries. You cannot swap MPI implementations without relinking. The three common choices are Intel MPI, OpenMPI, and Platform MPI (now part of Altair). Each has different rank-binding behavior, and rank binding directly affects crash simulation performance because memory locality determines how fast contact and element data are exchanged.

A validated MPI configuration for a dual-socket node with 64 cores per socket might use:

# Intel MPI rank binding for a 2-socket, 128-core node
export I_MPI_FABRICS=shm:ofi
export I_MPI_OFI_PROVIDER=mlx
export I_MPI_PIN=1
export I_MPI_PIN_DOMAIN=core
export I_MPI_PIN_ORDER=compact
mpirun -np 128 -ppn 128 ./lsdyna_mpp_d i=crash_deck.k ncpu=128

The compact binding order fills cores on socket 0 before moving to socket 1, which is usually correct for crash workloads because it keeps MPI ranks close to their memory. The scatter order is used when memory bandwidth per socket is the bottleneck. The choice must be validated with a reference deck, not assumed.

Validation Gate: The Reference Crash Deck

No node enters production until it passes a reference crash deck — a small but representative model (e.g., a 500k-element frontal impact) that exercises the full stack: MPI communication, GPU offload, filesystem I/O, and solver numerics. The acceptance criteria are:

Common Failure Patterns and Diagnosis

Three failure patterns account for most provisioning incidents in crash clusters:

1. MPI version mismatch after a partial update. A node gets a newer OpenMPI from a routine dnf update while the solver was linked against the older one. Symptom: mpirun starts but ranks hang at initialization. Diagnosis: compare mpirun --version across nodes and against ldd output on the solver binary.
2. GPU driver drift. A node reboots and picks up a newer driver from a repository. Symptom: CUDA error at solver startup. Diagnosis: nvidia-smi driver version vs. the baseline manifest.
3. OFED userspace/kernel mismatch. Symptom: intermittent MPI hangs under load, no error message. Diagnosis: ofed_info -s vs. modinfo ib_core version, and check HCA firmware with ibstat.

Deliverable: Node Acceptance Checklist

The following checklist is the minimum gate for admitting a node to the crash partition. It is designed to be filled per node and retained as an audit record.

Provisioning is complete only when every node in the partition passes all ten items. The image is then frozen, versioned, and stored alongside the baseline manifest. Any future change — a driver update, a kernel patch, a new solver build — triggers a new image version and a full re-validation. This discipline is what separates a crash cluster that produces trustworthy results from one that produces plausible-looking numbers.