4. Strong scaling experiments

Strong scaling answers a deceptively simple question: if I keep the model exactly the same and add more compute nodes, how much faster does the job actually run? In LS-DYNA performance engineering, this is the first experiment you should run on any new cluster or any new model, because it establishes the practical ceiling of parallel efficiency for that specific combination of mesh, contact topology, solver settings, and interconnect. The theoretical ideal is linear: doubling the node count halves the wall-clock time. Reality never matches that ideal, and the gap between measured and ideal scaling is where all the useful engineering information lives.

Defining the Experiment

A strong scaling study holds the problem size constant. The deck, mesh density, material models, contact definitions, termination time, and output intervals do not change between runs. Only the number of MPI ranks (and therefore nodes) increases. This is fundamentally different from weak scaling, where the problem grows with the machine. Strong scaling exposes communication overhead, load imbalance, and serial bottlenecks because the work per rank shrinks while the coordination cost per rank stays roughly constant or grows.

Before launching anything, fix the measurement protocol. Use the same LS-DYNA executable and version across all runs, the same compiler and MPI library, and the same ncpu decomposition strategy. Record the wall-clock time from the LS-DYNA timing summary, not the elapsed time from your job scheduler, because the scheduler includes queue wait and file staging. The relevant line in the d3hsp file or the terminal output is the total elapsed time reported at the end of the run, typically labeled Total under the timing block. Also capture the initialization time separately, since it does not scale and can dominate at high node counts.

Running the Sweep

A practical strong scaling sweep for a mid-size crash model might use 1, 2, 4, 8, 16, 32, and 64 nodes, each with a fixed number of ranks per node. For a 64-core-per-node system, that means 64, 128, 256, 512, 1024, 2048, and 4096 MPI ranks. Run each configuration at least twice to check variance; a single run can be misleading if the interconnect is shared or if the file system is busy.

Here is a representative submission pattern using a scheduler-agnostic MPI launch, with the decomposition controlled explicitly:

# Strong scaling sweep: fixed deck, increasing ranks
# Each run uses the same input file crash_v3.k
for NODES in 1 2 4 8 16 32 64; do
  RANKS=$((NODES * 64))
  mpirun -np ${RANKS} \
    -hostfile hosts_${NODES} \
    ls-dyna ncpu=${RANKS} i=crash_v3.k \
    memory=2000m \
    > run_${RANKS}.log 2>&1
done

After the sweep, extract the total wall-clock time from each log. The LS-DYNA timing summary breaks the run into phases: input processing, initialization, and the solve loop. For strong scaling analysis, use the solve-loop time or the total time consistently. If initialization is a large fraction, report both, because initialization scaling behavior is different and often worse.

Computing Speedup and Efficiency

With times in hand, compute two metrics. Speedup is S(N) = T(1) / T(N), where T(1) is the single-node time and T(N) is the time at N nodes. Parallel efficiency is E(N) = S(N) / N, expressed as a percentage. Ideal strong scaling gives S(N) = N and E(N) = 100%. In practice, efficiency drops as N grows.

Consider a hypothetical example to make the arithmetic concrete. Suppose a 2-million-element crash model runs in 7200 seconds on one node. The table below shows a plausible measured sweep, with speedup and efficiency computed from those times. These numbers are illustrative, not measured from a specific system.

The efficiency curve tells the story. Up to 8 nodes, efficiency stays above 80%, meaning the model parallelizes well. Beyond 16 nodes, the curve bends sharply. At 64 nodes, you are spending more than two-thirds of the machine's capacity on overhead. The practical question is not "what is the maximum node count?" but "where is the knee of the curve?" — the point beyond which adding nodes yields diminishing returns.

Diagnosing Deviation from Ideal

When efficiency falls, the cause is almost always one of four things: communication overhead, load imbalance, serial sections, or memory bandwidth saturation. The LS-DYNA timing summary gives you clues for each.

Communication overhead shows up as a growing fraction of time in the message-passing routines. In the timing summary, look for the MPI or communication time. If it grows faster than the solve time shrinks, the interconnect is the bottleneck. On InfiniBand or similar high-speed fabrics, this is less severe; on Ethernet, it can be brutal. The fix is usually to reduce the number of contact interfaces or to use a decomposition that minimizes inter-domain boundaries.

Load imbalance appears when some ranks finish their work long before others. LS-DYNA reports the per-rank timing if you set ncpu and enable verbose output. If the maximum rank time is significantly higher than the average, the decomposition is uneven. This is common with adaptive meshing, erosion, or contact-heavy regions that cluster on a few ranks. The remedy is to adjust the decomposition strategy or use a different ncpu value that produces a more balanced partition.

Serial sections are parts of the code that cannot be parallelized: input processing, certain contact algorithms, and some material model evaluations. These show up as a fixed time cost that does not shrink with N. If your total time at 64 nodes is dominated by a serial section that took 30 seconds at 1 node, you will never go below 30 seconds no matter how many nodes you add. The timing summary's initialization and input phases reveal this.

Memory bandwidth saturation occurs when the problem per rank becomes so small that the solver spends more time waiting for memory than computing. This is a hardware limit, not a software one. The symptom is that efficiency drops even though communication and imbalance look fine. The only fix is to use fewer nodes or a larger problem.

Practical Decision Making

The output of a strong scaling study is not a single number but a curve and a recommendation. For the illustrative example above, the knee is around 16 nodes, where efficiency is 70%. If your deadline allows, running at 16 nodes gives you a good balance of speed and resource efficiency. Running at 64 nodes is 20 times faster than one node but consumes 64 times the resources for that speedup. Whether that trade is acceptable depends on your queue priorities, allocation budget, and deadline pressure.

Finally, document the sweep. Record the node count, rank count, wall time, speedup, efficiency, and any notes about the decomposition or contact settings. A simple CSV or spreadsheet is enough. This record becomes the reference for future capacity planning and for detecting regressions when you upgrade the cluster or the solver. Strong scaling is not a one-time measurement; it is a baseline that you revisit whenever the hardware or the model changes.