4. Job scheduling and resource management for crash simulation workloads

Crash simulation is one of the most resource-hungry workloads in automotive CAE. A single full-vehicle frontal impact model may contain 8–15 million elements, run for 120–180 ms of physical time, and consume 2,000–20,000 core-hours depending on solver, contact complexity, and timestep. Multiply that by hundreds of load cases (frontal, offset, side, rear, pole, pedestrian, roof crush, battery intrusion) across dozens of vehicle programs, and the platform quickly becomes a scheduling problem rather than a solver problem. This chapter explains how batch systems, queues, and priority policies are structured to keep a crash simulation platform both fair and fast.

Why Crash Workloads Need a Batch Scheduler

Crash jobs are long-running, memory-heavy, and bursty. A typical LS-DYNA MPP job on 128 cores may hold 200–400 GB of RAM and run for 6–24 hours. Interactive submission is impractical: engineers submit jobs, disconnect, and expect results hours later. A batch scheduler (Slurm, PBS Pro, LSF, or SGE) provides three essential capabilities:

• Resource matching — allocating the exact core count, memory, and license tokens a job requests.
• Queueing — holding jobs when resources are unavailable and dispatching them in a controlled order.
• Policy enforcement — applying fair-share, priority, and preemption rules so no single team monopolizes the cluster.

Without a scheduler, a handful of 512-core jobs would starve the entire cluster, and license tokens for LS-DYNA or Pam-Crash would be consumed by jobs that cannot actually start.

Queue Design for Crash Simulation

Queues are not just FIFO lines; they are policy containers. A practical crash platform typically defines queues by job size and urgency rather than by department:

Separating debug from standard is critical: a 16-core debug job should never wait behind a 1,024-core production run. Small queues with short walltime limits drain quickly and keep iteration loops fast.

Priority Policies: Fair-Share, Age, and Preemption

Slurm's multifactor priority plugin is the de facto standard. The priority score is a weighted sum:

Priority = SiteFactor
         + (AgeFactor      × AgeWeight)
         + (FairShareFactor × FairShareWeight)
         + (JobSizeFactor  × JobSizeWeight)
         + (PartitionFactor × PartitionWeight)
         + (QOSFactor      × QOSWeight)

For crash workloads, a workable weighting is:

• Fair-share weight 0.5 — prevents one CAE team from consuming the cluster for a week.
• Age weight 0.3 — ensures long-waiting jobs eventually rise to the top.
• Job-size weight 0.1 — slight preference for smaller jobs to improve backfill.
• QOS weight 0.1 — allows a urgent QOS for safety-critical recalls.

Fair-share is computed from historical usage over a decay half-life (typically 7–14 days). A team that has consumed 3× its share will see its fair-share factor drop toward zero, letting other teams' jobs jump ahead.

License-Aware Scheduling

Crash solvers are license-bound as well as core-bound. LS-DYNA MPP tokens, Pam-Crash licenses, and HyperWorks solver tokens are finite. A job that requests 512 cores but cannot obtain 512 tokens will fail at startup. Two mechanisms solve this:

• License forwarding — the scheduler queries the license server (FlexNet) before dispatch and holds the job if tokens are unavailable.
• Token-per-core ratio — define a resource lstc_token with a 1:1 or 1:4 ratio to cores, and let the scheduler allocate both atomically.

# Slurm: request 256 cores and 256 LS-DYNA tokens
sbatch --nodes=4 --ntasks-per-node=64 \
       --gres=lstc_token:256 \
       --partition=standard \
       --time=24:00:00 \
       run_crash.sh

Preemption and Checkpointing

Preemption lets high-priority jobs reclaim resources from lower-priority ones. For crash simulation, preemption is expensive because restarting a 12-hour job from scratch wastes the entire run. Two strategies mitigate this:

1. Graceful preemption with SIGTERM — the scheduler sends SIGTERM, the solver writes a restart file (d3dump for LS-DYNA), and the job requeues. Requires --requeue and solver restart support.
2. Suspend/resume — the job is paused (SIGSTOP) and resumed later. Only viable for short preemptions; memory remains pinned.

In practice, most crash platforms disable preemption for large and standard queues and reserve preemption for the debug queue only, where jobs are short enough that restart cost is negligible.

Resource Accounting and Chargeback

Every job should record core-hours, memory-hours, and license-hours. These metrics drive fair-share, capacity planning, and chargeback. A minimal accounting record includes:

Putting It Together: A Scheduling Decision Flow

The diagram below shows how a submitted crash job moves through the scheduler: resource request validation, license check, queue selection, priority computation, backfill opportunity, and final dispatch or hold.

Practical Tuning Checklist

When a crash simulation platform underperforms, the bottleneck is usually scheduling policy, not hardware. Work through these checks in order:

1. Measure queue wait time by partition. If standard jobs wait >4 h on average, the partition is undersized or fair-share is too aggressive.
2. Check backfill effectiveness. scontrol show config | grep Backfill — if BackfillScheduler is not sched/backfill, enable it.
3. Audit CPU efficiency. Jobs below 60% efficiency waste cores; re-tune decomposition or reduce core count.
4. Verify license gating. If jobs start but immediately fail with license errors, the scheduler is not checking tokens before dispatch.
5. Review fair-share decay. A half-life that is too short causes thrashing; too long lets one team dominate. 7–14 days works for most CAE organizations.

With these policies in place, a well-tuned crash simulation platform can sustain 85–92% cluster utilization while keeping median queue wait under 30 minutes for standard load cases — the difference between a simulation team that iterates daily and one that iterates weekly.