2. Shared cluster architecture for crash CAE

Crash CAE workloads are not a single job type. A full-vehicle frontal impact model may run for 20–40 hours on 64–256 cores with hundreds of gigabytes of solver scratch, while a subsystem sled model or a design-of-experiments (DOE) sweep may consist of dozens of short 30-minute runs. A shared cluster must host both patterns, plus the pre/post-processing, meshing, and optimization traffic that surrounds them. This chapter describes the architectural layers that make that possible: the physical HPC topology, the batch scheduler that arbitrates resources, the node partitioning scheme that separates incompatible workloads, the interconnect that carries MPI traffic, and the storage tiers that absorb solver scratch I/O.

HPC topology: what the scheduler actually sees

A crash CAE cluster is normally organized as a set of racks containing compute nodes, a management/head tier, a parallel filesystem, and one or more login or submission hosts. Each compute node exposes CPU sockets, memory channels, local NVMe scratch, and one or two HCA ports. The scheduler does not see "a cluster"; it sees a list of nodes with attributes: CPU count, memory, GRES (generic resources such as GPUs), features, and partition membership.

Two topology facts dominate crash solver performance. First, NUMA locality: on a dual-socket node, memory attached to socket 0 is roughly 1.5–2× faster to access from socket 0 than from socket 1. LS-DYNA and Pam-Crash scale best when MPI ranks are bound to cores and memory is allocated locally. Second, fabric distance: a rank pair on the same leaf switch has far lower latency than a pair crossing spine switches. For explicit crash solvers, communication is dominated by nearest-neighbor domain-boundary exchange, so a decomposition that keeps adjacent domains on adjacent nodes reduces interconnect pressure.

Batch schedulers: Slurm, PBS Pro, and LSF

All three schedulers solve the same problem — match jobs to resources under policy — but their configuration models differ in ways that matter for crash CAE.

For crash CAE, the critical scheduler features are licence-aware admission (LS-DYNA, Pam-Crash, and Abaqus tokens are finite and expensive), topology-aware placement, and job arrays for DOE sweeps. A representative Slurm submission for a 128-core crash run looks like this:

#!/bin/bash
#SBATCH --job-name=front_impact_64core
#SBATCH --partition=crash
#SBATCH --nodes=4
#SBATCH --ntasks-per-node=32
#SBATCH --cpus-per-task=1
#SBATCH --mem=0
#SBATCH --gres=license:lsdyna:128
#SBATCH --time=24:00:00
#SBATCH --output=%x_%j.out

module load ls-dyna/13.1.0
export LSTC_LICENSE=network
export LSTC_LICENSE_SERVER=27000@lic-srv

srun --cpu-bind=cores --distribution=block:block \
  ls-dyna i=front_impact.key ncpu=128 memory=2000m

The --distribution=block:block directive packs consecutive ranks onto consecutive nodes, which keeps neighboring domains close on the fabric. The --gres=license:lsdyna:128 line makes the scheduler hold 128 tokens before the job starts, preventing the classic failure mode where a job launches, consumes 128 cores, and then dies because the licence server is exhausted.

Node partitioning

Partitioning is the primary isolation mechanism on a shared cluster. A workable scheme for crash CAE separates nodes by memory footprint, interconnect generation, and workload class:

• crash-large — 512 GB+ memory, HDR InfiniBand, reserved for full-vehicle models requiring 64–256 cores.
• crash-sweep — 128–256 GB memory, HDR, tuned for many concurrent 16–32 core DOE jobs.
• prepost — high clock, GPU-equipped, for HyperMesh, ANSA, and post-processing.
• debug — 2–4 nodes with short time limits for interactive solver testing.

Partitions should overlap only where policy allows. A common mistake is to let DOE sweeps flood the large-memory partition, fragmenting it so that a 256-core job waits for hours. Slurm QOS with MaxTRESPerUser and partition AllowGroups prevents this.

Interconnect: InfiniBand and RoCE

Crash solvers exchange boundary data every timestep. For a 10-million-element model on 128 ranks, per-timestep communication is typically tens of megabytes; at 1 ms timesteps, that is tens of GB/s of aggregate fabric traffic. InfiniBand HDR (200 Gb/s per port) or NDR (400 Gb/s) is the standard choice. RoCEv2 over 100/200 GbE is viable when the fabric is lossless (PFC + ECN configured) and the NICs support hardware offload; without lossless configuration, RDMA retransmits cause latency spikes that stall the solver.

Key checks before trusting a fabric for crash CAE:

# InfiniBand link and error counters
ibstat | grep -E "Rate|State|Physical"
perfquery -x 1        # symbol error and link downed counters

# RDMA bandwidth smoke test between two nodes
ib_write_bw -d mlx5_0 -a -F --report_gbits

# RoCEv2: verify PFC and ECN are active
mlnx_qos -i ens1f0    # shows PFC priorities and trust mode

If perfquery shows rising symbol errors or ib_write_bw delivers well below line rate, the solver will show erratic timestep times even though CPU utilization looks normal. That signature — high CPU, low effective scaling — is the classic fabric-degradation symptom.

Storage tiers for solver scratch I/O

Crash solvers write two very different I/O streams: the d3plot / state file output (sequential, moderate size, must survive the job) and scratch / restart data (large, bursty, node-local or parallel). A three-tier layout works well:

Point the solver's scratch directory at node-local NVMe via $TMPDIR or the solver's own scratch keyword, and write only d3plot to the parallel filesystem. This removes the metadata storm that occurs when hundreds of ranks open and close files on Lustre simultaneously.

# Slurm: give each job a private NVMe scratch directory
export TMPDIR=/nvme/scratch/$SLURM_JOB_ID
mkdir -p $TMPDIR
# LS-DYNA scratch keyword in the input deck:
# *CONTROL_TERMINATION ... 
# *DATABASE_BINARY_D3PLOT with a path on the parallel FS

Putting the layers together

The diagram below shows how a crash CAE job traverses the architecture: submission from a login node, admission by the scheduler with licence and partition checks, placement onto crash-partition nodes, MPI communication over InfiniBand, and I/O split between node-local NVMe scratch and the parallel filesystem for d3plot output.

The architecture is not a static diagram; it is a set of constraints that determine whether a crash job finishes in 18 hours or 40. When a job underperforms, the diagnostic order is consistent: check CPU binding and NUMA locality first, then fabric error counters and bandwidth, then scratch I/O placement, and only then revisit solver decomposition. Each layer has a measurable signature, and the scheduler configuration determines which layers a given job can even reach.