2. Parallel computing foundations for CAE

Before tuning a single solver parameter, a CAE engineer must answer a more fundamental question: how will this simulation actually be parallelized? The answer determines which hardware to buy, how to decompose the mesh, where the bottlenecks will appear, and whether adding cores will help at all. This section builds the conceptual foundation — shared-memory and distributed-memory models, Amdahl's and Gustafson's laws, and the scalability metrics used to judge real simulation performance.

Two Memory Models, Two Programming Realities

In the shared-memory model, all processing cores access a single global address space. A thread running on core 3 can read a variable written by a thread on core 7 without any explicit message passing. OpenMP is the canonical implementation: a serial code is annotated with directives such as #pragma omp parallel for, and the runtime distributes loop iterations across threads. The hardware substrate is typically a single multicore CPU socket or a multi-socket node with cache-coherent interconnects (Intel UPI, AMD Infinity Fabric). The critical performance hazard is false sharing — two threads writing to distinct variables that happen to reside on the same 64-byte cache line, causing the line to ping-pong between caches. In CAE kernels, this frequently appears when threads accumulate partial sums into adjacent array elements.

In the distributed-memory model, each process owns a private address space. Data exchange happens exclusively through explicit messages, most commonly MPI (MPI_Send, MPI_Recv, MPI_Allreduce, MPI_Isend/MPI_Irecv). A CAE mesh is partitioned into subdomains, one per rank. Elements on subdomain boundaries require ghost (halo) layers — copies of neighbor-owned nodes and elements — that are refreshed each time step or solver iteration. The cost of that refresh is the communication overhead, and it grows with the ratio of surface area to volume of each subdomain. This is why a 1-million-element mesh split across 8 ranks scales well, but the same mesh split across 512 ranks may spend more time in MPI_Allreduce than in floating-point work.

Modern HPC clusters combine both: each node runs one MPI rank per socket (or per NUMA domain), and within each rank OpenMP threads exploit the shared-memory cores. This hybrid MPI+OpenMP layout is the default for most implicit FE and CFD solvers.

Amdahl's Law: The Serial Fraction Ceiling

Amdahl's law answers a pessimistic question: given a fixed problem size, what is the maximum speedup achievable as the core count grows? If a fraction s of the runtime is inherently serial (mesh I/O, direct sparse factorization setup, global reductions that cannot overlap), then with N processors:

S(N) = 1 / (s + (1 - s) / N)

As N → ∞, speedup approaches 1/s. A code with just 5% serial content caps at 20× speedup no matter how many cores you throw at it. In CAE practice, the serial fraction often hides in places engineers overlook: assembling the global sparse matrix pattern, partitioning the mesh with METIS, writing restart files, or a single-threaded preconditioner setup. Profiling with tools like Intel VTune or Score-P to identify the actual serial fraction is far more reliable than assuming it is negligible.

Gustafson's Law: Scaling the Problem, Not Just the Cores

Gustafson's law takes the opposite view: in real CAE work, engineers rarely fix the problem size and add cores. They fix the time budget and increase the mesh resolution. If the serial fraction s stays constant while the parallel work grows with N, the scaled speedup is:

S_scaled(N) = s + N · (1 - s)

This is nearly linear for small s. The practical implication: a solver that looks like it "only" achieves 12× on 16 cores under Amdahl's fixed-size test may deliver 60× on 64 cores when the mesh is refined proportionally. This is why strong scaling and weak scaling are reported separately.

Scalability Metrics in Simulation Practice

Three metrics dominate CAE scalability reporting:

• Speedup S(N) = T₁ / T_N — wall-clock time on 1 core divided by wall-clock on N cores, same problem.
• Parallel efficiency E(N) = S(N) / N — fraction of ideal linear gain actually realized. Above 0.80 is generally considered good for implicit FE; below 0.50 signals a communication or load-balance problem.
• Weak scaling efficiency — fix work per core, grow both problem and core count. Ideal is a flat line at 1.0. A downward slope indicates that ghost-layer exchange or global reductions are growing faster than local work.

Reading a Scaling Curve: A Worked Example

Suppose a transient thermal simulation on a 2-million-element mesh is run on 1, 8, 32, and 128 cores. The measured wall-clock times are 7200 s, 980 s, 310 s, and 145 s. The computed metrics are:

The efficiency drop from 0.92 to 0.39 between 8 and 128 cores is the classic signature of a growing surface-to-volume ratio in the subdomains. At 128 ranks, each subdomain has roughly 15,600 elements but a disproportionately large boundary. The fix is not more cores — it is either a coarser partition (fewer, larger subdomains), a switch to a hybrid MPI+OpenMP layout with 4–8 threads per rank, or overlapping communication with computation using non-blocking MPI_Isend/MPI_Irecv.

Choosing the Right Model for Your Solver

Shared-memory parallelism is simplest to implement and has the lowest communication latency, but it is bounded by the core count of a single node. Distributed-memory parallelism scales to thousands of nodes but demands explicit data management and pays a latency penalty on every message. The pragmatic default for large CAE simulations is hybrid: MPI across nodes, OpenMP within each node, with the rank-to-core ratio tuned so that each rank has enough work to amortize message latency. The scaling laws and metrics above give you the vocabulary to measure whether that tuning actually worked — and to know when the answer is "stop adding cores and fix the decomposition instead."