4. Model preparation for parallel benchmarking

Before a single MPI rank is launched, the quality of a PAM-CRASH parallel benchmark is largely determined by how the finite element model is decomposed. PAM-CRASH uses explicit time integration, so every cycle requires element-level force computation followed by nodal force summation and communication of interface contributions between subdomains. If the mesh is partitioned poorly, one rank becomes the critical path while the others idle at the next MPI barrier. This section explains how PAM-CRASH performs domain decomposition, how to prepare a model that partitions cleanly, and how to verify that the resulting load balance is actually suitable for benchmarking.

How PAM-CRASH Decomposes a Model

PAM-CRASH does not partition the model at runtime inside the solver. Instead, the model is decomposed in a preprocessing step, typically through the PAM-CRASH domain decomposition utility or an external partitioner invoked from the solver input deck. The decomposition produces one subdomain input file per MPI rank, plus a communication table describing which nodes are shared between which ranks. At runtime, each rank reads only its own subdomain and exchanges boundary nodal data with its neighbors through MPI.

Two decomposition strategies are common in PAM-CRASH workflows:

  • Geometric partitioning — the mesh is cut by spatial planes or recursive coordinate bisection (RCB). This is fast and robust, but it ignores element connectivity and can produce many cut elements and a large interface.
  • Topological partitioning — graph-based methods such as METIS or ParMETIS minimize the number of cut edges while balancing element counts. This is the preferred approach for crash models with irregular geometry, because it reduces both communication volume and the number of interface nodes.

The choice matters because PAM-CRASH communication cost scales with the number of shared nodes and the number of neighboring subdomains, not just with element count. A partition with balanced element counts but a fragmented interface can be slower than a slightly imbalanced partition with a compact interface.

PAM-CRASH Domain Decomposition Workflow Input Model nodes, shells, solids contacts, materials loads, BCs Partitioner METIS / ParMETIS or RCB geometric target: N ranks Subdomain Files rank_0000.inp rank_0001.inp ... rank_N.inp MPI Runtime ghost exchange force summation contact search Load Balance Metrics (example) Rank Elements Interface Nodes Neighbors 0 124,800 8,420 4 1 125,100 8,390 4 2 124,950 8,510 5 3 168,400 14,200 7 Imbalance ratio (max/avg): 1.34 Rank 3 is critical path — rebalance required Target: imbalance < 1.05, neighbors ≤ 6 Interface nodes should be < 10% of local nodes Partition Quality Comparison Topological (METIS) A B C D E F Compact interface, balanced Geometric (RCB) A B C D E F G H I J K L Fragmented interface, imbalanced

Preparing the Model for Clean Partitioning

Partition quality is not only a function of the partitioner. The input model itself must be prepared so that the decomposition has a chance to succeed. The following preparation steps are standard in PAM-CRASH benchmarking practice.

1. Remove unused and orphan entities. Free nodes, unused materials, and disconnected shell patches inflate the graph and confuse the partitioner. Run a model check to delete unreferenced nodes and elements before decomposition. In PAM-CRASH, this is typically done with the model checking options in the preprocessor or with a dedicated cleanup pass.

2. Merge coincident nodes. Duplicate nodes at material interfaces or from imported CAD geometry create artificial graph edges and increase interface size. Use a tolerance-based merge (for example, 0.01 mm for a typical automotive crash model) and verify that the merge does not collapse thin features.

3. Handle contact definitions explicitly. PAM-CRASH contact interfaces are not automatically partitioned with the mesh. Self-contact and tied contact definitions must be split across subdomains, and the partitioner must be told which contact surfaces belong to which subdomain. If a contact surface spans many subdomains, the contact search cost grows and can dominate the runtime. Where possible, define contact surfaces so that they align with expected subdomain boundaries.

4. Check element quality and connectivity. Negative-volume solids, collapsed shells, and non-manifold edges cause the partitioner to produce degenerate subdomains or fail outright. Run a quality check and fix or remove bad elements before decomposition.

5. Decide the target rank count from the model size. A practical starting rule for explicit crash models is 50,000–150,000 elements per rank on a modern HPC node. For a 4-million-element model, that suggests 32–64 ranks. The benchmark should then sweep around this range (for example, 16, 32, 64, 128 ranks) to find the scaling knee.

Load Balancing: What to Measure and How to Judge It

After decomposition, the subdomain files should be inspected before running the solver. The key metrics are:

MetricWhat it indicatesPractical target
Element count per rankComputational balanceMax/avg ratio < 1.05
Interface (ghost) nodes per rankCommunication volume< 10% of local nodes
Number of neighboring ranksMPI message count and latency exposure≤ 6 for most ranks
Contact surface distributionContact search imbalanceNo rank owns > 2× average contact segments
Critical path rankWall-clock bottleneckIdentified and rebalanced

If the element count imbalance exceeds roughly 1.10, the partitioner should be rerun with a tighter balance constraint or a different method. METIS, for example, exposes a ufactor parameter that controls the allowed imbalance; reducing it forces a more even distribution at the cost of a larger interface. In PAM-CRASH workflows, this is often controlled through the decomposition input rather than the solver deck.

A useful diagnostic is to plot per-rank element count and interface node count as a bar chart. A single tall bar immediately identifies the critical path rank. If that rank also has many neighbors, the problem is both computational and communicational, and a topological repartition is usually the right fix.

Domain Decomposition in Practice: A Reproducible Example

Consider a full-vehicle frontal impact model with approximately 3.2 million elements, including 1.8 million shells, 1.2 million solids, and 200,000 beam/spring elements. The benchmark target is a 64-rank run on a cluster with 64 cores per node.

Step 1 — Clean and merge. Run the model check, delete 4,200 orphan nodes, and merge coincident nodes with a 0.01 mm tolerance. This reduces the node count by about 1.5% and removes artificial graph edges at material interfaces.

Step 2 — Decompose with METIS. Invoke the partitioner with 64 subdomains and a balance constraint of 1.03. The output is 64 subdomain files plus a communication map. Example command line (illustrative):

pamcrash -decomp -nproc 64 -method metis -ufactor 1.03 \
         -model vehicle_full.inp -out subdomains/

Step 3 — Inspect the decomposition report. The report lists per-rank element counts, interface nodes, and neighbor counts. In this example, the initial METIS run produces a max/avg element ratio of 1.04 and an average of 4.2 neighbors per rank, with one rank owning 12% of the contact segments. That contact imbalance is the main risk.

Step 4 — Rebalance contact. Redefine the contact surface grouping so that the large self-contact on the underbody is split into two surfaces, each assigned to a different subdomain cluster. Rerun the decomposition. The contact segment distribution now has a max/avg ratio of 1.08, which is acceptable for a first benchmark pass.

Step 5 — Verify with a short run. Run 50 explicit cycles with the 64-rank decomposition and collect per-rank wall-clock time for element processing, communication, and contact search. If one rank consistently spends more than 20% above the mean in any category, the decomposition should be revisited before the full benchmark.

Do not assume that a balanced element count guarantees a balanced runtime. In explicit crash simulations, contact search and element formulation cost vary significantly between shell-heavy and solid-heavy regions. A rank with fewer elements but a dense contact patch can still be the critical path. Always validate with a short instrumented run before committing to a full benchmark matrix.

Common Failure Symptoms and Troubleshooting

  • Partitioner fails with "graph is disconnected." The model contains isolated nodes or elements. Run a connectivity check and remove or reconnect orphan entities.
  • One rank runs out of memory. The decomposition is imbalanced, or a single rank owns a disproportionately large contact surface. Check the per-rank element and contact counts, and repartition with a tighter balance constraint.
  • Communication time dominates. The interface is too large or the neighbor count is too high. Switch from geometric to topological partitioning, or increase the number of ranks to reduce per-rank interface size.
  • Results differ between serial and parallel runs. This usually indicates that contact definitions or tied interfaces were not partitioned consistently. Verify that the communication map includes all shared nodes and that contact surfaces are assigned to the correct subdomains.
  • Scaling stalls beyond a certain rank count. The model may be too small for the rank count, so per-rank work is dominated by communication and startup. Reduce the rank count or increase the model size for the benchmark.

In summary, model preparation for PAM-CRASH parallel benchmarking is a sequence of deliberate checks: clean the mesh, choose a partitioner that respects connectivity, decompose with an explicit balance target, inspect the per-rank metrics, and validate with a short instrumented run. The decomposition is not a one-time preprocessing detail — it is the foundation on which every subsequent scaling measurement depends.

← Previous 📖 Contents Next →