2. Legacy HPC cluster architecture in automotive CAE

Most automotive OEMs and Tier-1 suppliers built their first serious CAE capacity between roughly 2005 and 2015. Those clusters were designed around a simple premise: run a large number of independent, single-job-per-node simulations — crash, NVH, CFD, durability — and let a batch scheduler pack them onto whatever hardware was free. Understanding that legacy architecture matters because migration projects rarely start from a blank sheet. They start from a working cluster with a specific node generation, a specific interconnect, a specific parallel filesystem, and a scheduler whose queue policies encode a decade of organizational habits.

Compute Nodes: The Building Block

A typical legacy CAE compute node in this era was a dual-socket x86 server. A representative configuration might be two 12-core Intel Xeon E5-2680 v3 processors (24 physical cores, 48 threads with HT), 128 GB DDR4-2133, a single 400 GB SAS SSD for scratch, and a Mellanox ConnectX-3 FDR InfiniBand HCA. Nodes were usually grouped into "islands" of 32–64 nodes connected to a leaf switch, with leaf switches uplinked to a spine layer.

Two characteristics of these nodes dominate migration planning. First, memory per core is low by modern standards — 128 GB / 24 cores ≈ 5.3 GB per core. Modern implicit crash models and CFD meshes routinely exceed that, which is why legacy clusters often show high swap activity or out-of-core solver behavior. Second, the CPU generation lacks AVX-512 and has limited memory bandwidth, so solver scaling curves flatten earlier than on current hardware. When you benchmark a candidate migration target, always compare per-core and per-node throughput separately, because licensing and queue policy are usually per-core while wall-clock cost is per-node.

Interconnect: Where Parallel Jobs Live or Die

Legacy automotive clusters almost universally used InfiniBand, most commonly FDR (56 Gb/s) or EDR (100 Gb/s), with some older sites still on QDR. The interconnect matters most for tightly coupled MPI jobs — explicit crash, CFD, and large NVH modal runs. A useful rule of thumb: if a job's MPI communication exceeds roughly 10–15% of runtime, interconnect latency and bandwidth become first-order. On FDR fabrics, MPI latency is typically 1.5–2.5 µs, and effective bandwidth per node is often 4–6 GB/s in practice, not the theoretical 6.8 GB/s.

Topology also matters. Legacy clusters frequently used a two-level fat-tree with 1:1 or 2:1 oversubscription at the spine. A 2:1 oversubscribed fabric can halve effective bandwidth when many nodes communicate across leaf boundaries. Before migrating, capture the actual job placement pattern: are jobs confined to a single island, or do they span the whole cluster? This determines whether the new fabric needs full bisection bandwidth or can tolerate oversubscription.

Storage Tiers: Scratch, Home, and Archive

Legacy CAE storage was typically three-tiered:

• Node-local scratch — SAS SSD or, in older sites, 15K RPM SAS disks. Used for solver temporary files, which can be tens to hundreds of GB per job.
• Central parallel filesystem — Lustre or GPFS/Spectrum Scale, often 500 TB–2 PB usable, with 4–8 OSTs or NSDs per couple hundred nodes. Metadata servers were frequently a bottleneck.
• Archive / nearline — tape (LTO) or a slower NAS tier for completed results and long-term retention.

The critical migration risk here is not capacity but metadata performance and small-file I/O. CAE workflows generate enormous numbers of small files: solver restart files, per-step result files, log files, and post-processing extracts. A legacy Lustre filesystem with 2 MDTs can struggle when a migration tool or a new scheduler starts stat-ing millions of files. Always measure metadata operations per second (stat, open, create) on the source before designing the target.

Schedulers and Queue Policy

Legacy clusters predominantly ran Altair PBS Professional, IBM Spectrum LSF, or SGE/Univa Grid Engine. Slurm was present but less common in older automotive sites. The scheduler is where organizational policy lives, and it is often the hardest thing to migrate because it encodes fairshare rules, project accounting, license token management, and preemption priorities.

A representative PBS queue structure might look like this:

# qstat -q
Queue            Memory CPU Time Walltime Node  Run Que Lm  State
---------------- ------ -------- -------- ---- --- -- --- -----
crash_short       --      --    12:00:00   --   40  12  40  E R
crash_long        --      --    72:00:00   --   20   3  20  E R
cfd_high          --      --    48:00:00   --   16   8  16  E R
nvh               --      --    24:00:00   --   24   5  24  E R
post              --      --    04:00:00   --    8   2   8  E R

Each queue maps to a project, a license pool, and a walltime limit. Migration to Slurm requires translating these into partitions, QOS definitions, and sacctmgr associations. The subtle part is fairshare: PBS uses a formula based on historical usage, while Slurm uses a decayed half-life model. A naive translation can silently change who gets priority, which is a political problem as much as a technical one.

Job Submission Workflow

The legacy submission workflow is usually a shell script wrapping a solver invocation, submitted with qsub or bsub. A minimal PBS example:

#!/bin/bash
#PBS -N crash_frontal
#PBS -q crash_short
#PBS -l select=4:ncpus=24:mpiprocs=24
#PBS -l walltime=08:00:00
#PBS -P project_abc
#PBS -o crash_frontal.log
#PBS -e crash_frontal.err

cd $PBS_O_WORKDIR
module load ls-dyna/971_R11.1
mpirun -np 96 ls-dyna i=frontal.key ncpu=96 memory=200m

Key details to capture during migration: the select syntax (chunk-based in PBS, node-based in Slurm), the module system (Environment Modules vs Lmod), the MPI launcher (mpirun vs srun), and the working directory convention. These four items account for the majority of submission-script breakage.

What to Capture Before Migration

Before touching the target environment, build a factual baseline of the legacy cluster. The following inventory is the minimum needed to size and validate a migration:

Collect this data with a combination of scheduler accounting logs (qstat -f, bhist, sacct), filesystem statistics (lfs df, mmlsfs), and a short benchmark campaign using representative production decks. The benchmark should include at least one tightly coupled MPI job, one memory-heavy implicit job, and one I/O-heavy job with many small result files. Run each on the legacy cluster and on the target, and compare wall-clock, CPU efficiency, and peak memory. That comparison — not vendor specifications — is the evidence base for the migration business case.