3. Licensing infrastructure for PAM-CRASH H

PAM-CRASH is a commercial explicit finite element solver, and every concurrent solver process consumes a license token from a FlexNet Publisher (FlexLM) license server. On a workstation this is trivial. On an HPC cluster running dozens of parallel jobs across hundreds of cores, licensing becomes a shared, stateful service that can silently throttle throughput, block job startup, or fail an entire batch queue. This section covers how to design, configure, and harden that service.

How PAM-CRASH Licensing Actually Works

ESI Group ships PAM-CRASH licenses through FlexNet Publisher. The vendor daemon is typically named esi_sld (older releases use pam_lmd), and it is managed by the standard lmgrd broker. The license file contains SERVER, VENDOR, and INCREMENT/FEATURE lines. A representative entry looks like this:

SERVER lic01.cluster.local 001122334455 27000
VENDOR esi_sld /opt/esi/license/esi_sld
USE_SERVER

INCREMENT PAM_CRASH esi_sld 2026.0310 31-dec-2026 50 \
  VENDOR_STRING=HPC DUP_GROUP=UHD ISSUED=10-jan-2025 \
  SN=12345678 SIGN="00A1 B2C3 ..."

The critical field is the token count — 50 in this example — which is the maximum number of concurrent tokens the daemon will grant. The DUP_GROUP=UHD attribute is equally important: it defines how tokens are shared when multiple processes originate from the same user, host, and display. For HPC batch workloads, DUP_GROUP=UHD means that all MPI ranks launched from the same job on the same node count as a single token, whereas DUP_GROUP=NONE would charge one token per rank and exhaust the pool almost immediately.

Server Configuration and Environment

A production license server should be a dedicated, low-latency host — not a login node and never a compute node. The daemon must be reachable on a fixed TCP port (commonly 27000 for lmgrd and a vendor port such as 27001 for esi_sld). Pin both ports explicitly in the license file and in the firewall rules; dynamic port allocation is a common source of intermittent checkout failures across subnets.

Client-side configuration is minimal but must be consistent across every compute node. Set the environment variable in the module file or job prologue:

export ESI_LICENSE_FILE=27000@lic01.cluster.local
# or, for multiple servers:
export ESI_LICENSE_FILE=27000@lic01:27000@lic02

Prefer ESI_LICENSE_FILE over the legacy LM_LICENSE_FILE to avoid collisions with other FlexLM-based applications on the same node. On the server side, start the daemon with an explicit log and debug log so that checkout denials are traceable:

/opt/esi/license/lmgrd -c /opt/esi/license/esi.lic \
  -l /var/log/flexlm/lmgrd.log \
  -2 -p -15

The -2 -p flags enable verbose logging, and -15 sets a 15-second retry interval when the vendor daemon dies. Run lmgrd under a service manager (systemd unit or a cluster resource agent) so it restarts automatically after a reboot or crash.

Token Management in Practice

Token exhaustion is the single most common licensing incident on a PAM-CRASH cluster. The diagnostic command is lmstat:

lmstat -a -c 27000@lic01.cluster.local

Typical output shows the feature name, total tokens, tokens in use, and the user/host holding each token. A healthy pool shows headroom; a saturated pool shows Users of PAM_CRASH: (Total of 50 licenses issued; Total of 50 licenses in use). When a job cannot check out a token, the solver exits with an error such as Cannot connect to license server or License error: -4,12547 (the exact code varies by release).

Two operational levers reduce contention without buying more licenses:

• Queue-level token gating. Configure the scheduler to reserve tokens before dispatching a job. In Slurm, a Licenses=pam_crash@lic01:1 resource request in the job script makes the scheduler hold the job until a token is free, rather than letting it start and fail.
• Token timeout tuning. The TIMEOUT and TIMEOUTALL directives in the license file reclaim tokens from crashed or orphaned processes. A value of TIMEOUT 3600 is a reasonable starting point for long explicit runs; too short a timeout will kill legitimate long-running jobs.

For accounting and chargeback, parse the lmgrd debug log or use lmstat -a snapshots into a time-series database. This reveals peak demand windows and justifies additional token purchases with real data rather than guesswork.

High-Availability Topologies

A single license server is a single point of failure: if it goes down, every queued and running job that needs a new token stalls. FlexNet supports three HA patterns, each with different trade-offs.

The three-server redundant configuration is the standard for production HPC. It requires three hosts (or three VMs on separate physical hardware) and a license file with three SERVER lines. The daemons form a quorum: any two can grant tokens, and the loss of one does not interrupt service. The client environment lists all three:

export ESI_LICENSE_FILE=27000@lic01:27000@lic02:27000@lic03

For a cold standby, keep the license file, daemon binaries, and startup scripts synchronized via configuration management (Ansible, Puppet). The standby host must have the same hostname resolution and firewall rules as the primary. Test failover quarterly by stopping the primary daemon and confirming that a test job can still check out a token.

Putting It Together

The diagram below summarizes the relationship between the license server tier, the scheduler, and the compute nodes, including the token checkout path and the HA failover path.

In this topology, the scheduler holds a job until a token is reserved, the compute nodes check out tokens directly from whichever license server answers first, and the three-server quorum tolerates the loss of any single license host. Combined with lmstat monitoring and log-based accounting, this gives a licensing tier that scales with the cluster rather than becoming its bottleneck.