4. License management for crash solvers

In an HPC production environment running LS-DYNA, PAM-CRASH, Radioss, or Abaqus/Explicit, the license pool is often the first resource to become scarce — long before CPU cores or memory. A 512-core cluster can be fully provisioned, yet a single 50-token LS-DYNA pool will throttle throughput to a fraction of that capacity. License management is therefore not an administrative afterthought; it is a scheduling constraint that must be modeled, monitored, and tuned with the same rigor as node allocation.

License Server Architecture and Redundancy

Most crash solvers use FlexNet Publisher (FlexLM) or a vendor-specific variant. The license server daemon (lmgrd) reads a license file, starts one vendor daemon per feature family (for example lstc for LS-DYNA, altaire or pam_lmd for PAM-CRASH), and listens on a fixed TCP port. Clients locate the server through the LM_LICENSE_FILE or LS_DYNA_LICENSE_FILE environment variable, or through a port@host string such as 29000@lic01.hpc.local.

Production deployments should use a three-server redundant configuration. FlexLM requires the three hosts to share a common triad license file and to be able to reach each other on both the lmgrd port and the vendor daemon port. A typical triad entry looks like:

SERVER lic01 001122334455 29000
SERVER lic02 001122334466 29000
SERVER lic03 001122334477 29000
VENDOR lstc /opt/lstc/lstc port=31000
USE_SERVER

The triad tolerates the loss of any single server. If two servers fail, the pool becomes unavailable and all running jobs that need to check out a new token will fail at their next license request. This is why the license servers should live on dedicated, monitored VMs or physical hosts — never on a login node that is rebooted for maintenance.

Token Allocation Models

Crash solvers typically license by one of three models, and the model determines how you schedule:

LS-DYNA SMP, for example, is commonly licensed per core, while MPP is licensed per core with a separate MPI feature. PAM-CRASH often uses a base token plus additional tokens for each extra core beyond a threshold. Before writing any scheduler policy, confirm the exact checkout behavior with lmstat -a -c 29000@lic01 and by running a controlled job at 8, 16, and 32 cores while watching the token count.

Queuing and Fair-Share Integration

The scheduler must know how many tokens are free before dispatching. Slurm exposes this through a Licenses resource in slurm.conf:

Licenses=lstc:200, pamcrash:120
PriorityWeightTRES=Licenses=1000

Jobs then request tokens explicitly:

#SBATCH --licenses=lstc:32
#SBATCH --nodes=2
#SBATCH --ntasks-per-node=16

With this configuration, Slurm will not start a job unless 32 lstc tokens are available, and it will hold the job in a LICENSE pending reason rather than letting it fail inside the solver. This is far preferable to letting LS-DYNA abort with "LS-DYNA license not available" after the job has already consumed queue time and staged input decks.

Fair-share should weight license consumption, not just CPU. A user running 64-core jobs consumes four times the tokens of a 16-core user; if the fair-share tree only counts CPUs, token-heavy users will starve the pool. Add a license TRES weight so that accounting reflects the real bottleneck.

Monitoring Utilization

Two metrics matter: instantaneous checkout count and checkout duration. A pool that is 100% utilized for 30-second bursts is healthy; a pool at 85% with multi-hour holds is a bottleneck. Poll lmstat every 60 seconds and write the parsed counts to a time-series store:

lmstat -a -c 29000@lic01 | awk '
  /Users of lstc/ {feature="lstc"}
  /^[[:space:]]+[0-9]+ licenses? in use/ {print feature, $1}
'

Feed the output into Prometheus via a textfile collector, then alert on two conditions: sustained utilization above 90% for more than 15 minutes, and any LICENSE-pending job older than 10 minutes. The first indicates the pool is undersized; the second indicates a scheduling or checkout leak.

Practical Tuning Workflow

1. Baseline: record peak concurrent tokens over one week of production.
2. Measure checkout duration distribution per feature.
3. Set Slurm Licenses to the true pool size, not the nominal count.
4. Add license TRES to fair-share and to priority weights.
5. Alert on sustained >90% utilization and on stale checkouts.
6. Re-baseline after any solver version change, since checkout behavior can shift.

Interpreting Utilization Data

A healthy crash-solver license pool typically shows a bimodal utilization curve: a daytime plateau near 70–85% and a nighttime plateau near 40–60% driven by batch sweeps. If the daytime plateau sits at 95%+ with a growing LICENSE pending queue, you have three options: purchase more tokens, reduce per-job core counts, or introduce a token-aware backfill policy that runs small jobs during gaps. The third option is often the cheapest and is implemented by setting a lower PriorityWeightTRES for large license requests so that small jobs can slip into the pool between long holds.

Finally, always correlate license utilization with job failure logs. A spike in "license not available" errors that does not match a utilization spike usually indicates a checkout leak or a misconfigured client environment variable, not a genuine shortage. Checking lmstat -a for users whose jobs have already exited is the fastest way to confirm this.