2. Cluster architecture for LS-DYNA

LS-DYNA MPP (Massively Parallel Processing) decomposes the model into domains and solves them concurrently across ranks. Because explicit time integration advances with a global minimum time step, every cycle requires at least one synchronization point between ranks. The cluster fabric, memory locality, and clock behavior therefore do not merely "support" scaling — they set its ceiling. This chapter examines the four layers that most often determine whether an LS-DYNA job scales linearly to 512 cores or stalls at 128.

Interconnect Options: InfiniBand vs. RoCE

InfiniBand (IB) is the reference fabric for LS-DYNA clusters. Modern HDR (200 Gb/s per port) and NDR (400 Gb/s) fabrics provide RDMA with kernel bypass, sub-microsecond switch latency, and hardware-level congestion control. LS-DYNA's MPI implementation (typically Intel MPI, Open MPI, or Platform MPI) maps its point-to-point and collective operations directly onto IB verbs, so message latency stays low even at high rank counts.

RoCE (RDMA over Converged Ethernet) delivers the same verbs API over Ethernet. RoCEv2 encapsulates IB transport over UDP/IP, which makes it routable but also introduces sensitivity to Ethernet's lossy behavior. The critical distinction is lossless configuration: RoCE requires Priority Flow Control (PFC) and Explicit Congestion Notification (ECN) to be tuned correctly on every switch and NIC. Without PFC, a single dropped packet triggers go-back-N retransmission and latency spikes that propagate through the collective. With PFC misconfigured, you can create head-of-line blocking or PFC storms that are harder to diagnose than simple packet loss.

For LS-DYNA specifically, the practical guidance is:

• Contact-heavy crash models generate bursty, small-message traffic. IB's hardware congestion control handles this better than RoCE unless PFC/ECN are carefully tuned.
• Large implicit or NVH runs with big sparse matrix operations benefit from high bandwidth; both fabrics can deliver, but RoCE's cost advantage is real at scale.
• Mixed workloads on a shared Ethernet fabric are risky for RoCE — storage traffic and LS-DYNA traffic compete for the same buffers.

NUMA Topology and Rank Placement

On a dual-socket node, each CPU has its own memory controller and LLC. A rank that allocates memory on the remote socket pays a latency penalty on every access — typically 1.5–2× local latency. LS-DYNA's explicit solver is memory-bandwidth-bound, so remote memory access directly reduces per-rank throughput.

The standard mitigation is one MPI rank per NUMA domain (not per core), with OpenMP threads filling the cores within that domain. For a 2×64-core node, that means 2 ranks × 64 threads, not 128 ranks. This keeps each rank's element and node arrays local to its socket.

# Example: 2-socket node, 64 cores/socket, bind ranks to NUMA nodes
mpirun -np 2 -ppn 2 \
  -genv I_MPI_PIN_DOMAIN numa \
  -genv I_MPI_PIN_ORDER compact \
  -genv OMP_NUM_THREADS 64 \
  -genv OMP_PROC_BIND close \
  ls-dyna_mpp_d R=deck.k NCPU=2 OMP=64

Verify placement with numactl --hardware and check that each rank's /proc/<pid>/numa_maps shows local allocation. If numa_hit is low relative to numa_miss, the rank is allocating remotely — usually because the MPI library's default binding is core rather than numa.

CPU Frequency Scaling and Turbo Behavior

LS-DYNA's explicit solver is a tight loop of floating-point operations with high memory traffic. It responds strongly to clock frequency but also generates enough power draw to trigger frequency throttling. Two BIOS settings matter:

• SpeedStep / P-states: must be disabled or set to "Performance" in BIOS. OS-level cpufreq governors set to powersave will drop cores to 800 MHz between MPI phases, adding latency to every synchronization.
• Turbo Boost: helpful for low core counts, but at high occupancy (all cores active) the package power limit forces a lower all-core frequency. A 3.5 GHz base / 4.2 GHz turbo CPU may run all-core at 3.0–3.2 GHz under sustained LS-DYNA load.

This creates a non-obvious scaling effect: adding cores increases total throughput but reduces per-core frequency, so the speedup curve flattens earlier than the core count alone would suggest. Measure actual frequency during a run with turbostat --interval 5 and correlate with LS-DYNA's cycle count output.

How These Layers Interact

The four layers form a dependency chain. NUMA misplacement increases memory latency, which lengthens each rank's compute phase, which makes the interconnect synchronization proportionally more expensive, which is then amplified by frequency throttling that slows the compute phase further. Fixing only one layer rarely recovers full scaling.

Practical Scaling Check

Before committing to a large core count, run a short benchmark on the actual deck. A reproducible procedure:

1. Run the deck at 32, 64, 128, and 256 ranks with identical NUMA binding and frequency settings.
2. Record Elapsed time from the LS-DYNA output and compute speedup S(n) = T(32) / T(n).
3. Compute parallel efficiency E(n) = S(n) / (n/32). Below 0.7 at 128 ranks signals a bottleneck in one of the four layers.
4. Cross-check with turbostat (frequency), numa_maps (locality), and ib_send_bw (fabric).

Typical failure signatures: efficiency drops sharply between 64 and 128 ranks with flat frequency → interconnect or collective bottleneck; efficiency degrades gradually with rising numa_miss → rank placement problem; efficiency drops only on long runs → thermal throttling. Each points to a different layer, and the fix is rarely "add more cores."