4. License server architecture and high availability

In a production CAE environment, the license server is a single point of failure for every solver job that depends on it. A structural analysis run in Abaqus, a CFD solve in Fluent, or a crash simulation in LS-DYNA cannot start without first checking out a feature token. If the license daemon is unreachable, jobs queue, batch schedulers stall, and engineering throughput collapses. This chapter examines how license server architectures are designed for high availability, how failover actually works at the protocol level, and what happens to solver job startup when a license server transitions between states.

Why Redundancy Is Not Optional

Most commercial CAE vendors use FlexNet Publisher (formerly FLEXlm) as their licensing backbone. FlexNet supports three server redundancy models: single server, redundant server pair, and triad. A single-server configuration is the simplest but offers no fault tolerance. A redundant pair uses two servers with a shared quorum, while a triad uses three servers and tolerates the failure of any one node. The choice depends on the criticality of the workload and the acceptable downtime window.

The key concept in FlexNet redundancy is the quorum. In a triad, each server runs lmgrd and the vendor daemon (for example, MLM for MSC Nastran or ansyslmd for Ansys). The three servers communicate over TCP and maintain a voting mechanism. A license checkout succeeds only if a majority of servers (two out of three) are reachable. This means a triad can survive one server failure without interrupting license service. A redundant pair, by contrast, requires both servers to be reachable for a checkout to succeed; if one fails, the pair loses quorum and no new licenses can be issued.

Critical distinction: A redundant pair provides high availability of the license file but not fault tolerance. Only a triad provides true fault tolerance. Many organizations mistakenly deploy a pair and assume they are protected against a single server outage.

Triad Configuration in Practice

Setting up a triad requires three physical or virtual servers, each with a static IP address and a consistent hostname resolution. The license file must list all three servers in the SERVER lines, followed by the vendor daemon line. A typical triad license file looks like this:

SERVER lic-srv-01 001122334455 27000
SERVER lic-srv-02 001122334466 27000
SERVER lic-srv-03 001122334477 27000
USE_SERVER
VENDOR ansyslmd PORT=1055

Each server runs lmgrd with the same license file. The USE_SERVER line tells the client to query the server for the vendor daemon port rather than hardcoding it. The triad uses port 27000 for lmgrd and a vendor-specific port (1055 in this example) for the daemon. Firewalls must allow bidirectional TCP traffic on these ports between all three servers and all compute nodes.

When a triad starts, the servers elect a master. The master handles all license checkouts and writes the license state to a shared journal. The other two servers act as replicas, receiving state updates. If the master fails, the remaining two servers hold an election and promote a new master within seconds. Clients that have already checked out licenses continue to hold them, but new checkouts are briefly paused during the election.

Failover Behavior and Solver Job Startup

The impact of failover on solver job startup depends on the timing and the client's retry configuration. When a compute node requests a license, the FlexNet client library sends a request to the master server. If the master is unreachable, the client times out after a configurable interval (default 30 seconds) and retries. In a triad, the client may also query the other servers directly if the master does not respond.

Consider a concrete example. A cluster runs 200 Ansys Fluent jobs. Each job requests 4 fluent tokens at startup. The license server is a triad. At 10:00:00, the master server (lic-srv-01) suffers a network partition. The remaining two servers detect the loss of heartbeat within 5 seconds and elect a new master (lic-srv-02). During this 5-second window, any job that attempts a checkout receives a "cannot connect to license server" error. The FlexNet client retries after 30 seconds. By then, the new master is active, and the checkout succeeds. The net effect is a 30–35 second delay for jobs that hit the window, but no job fails permanently.

If the same scenario occurs with a redundant pair, the outcome is different. When one server of a pair fails, the pair loses quorum. The surviving server cannot issue new licenses because it cannot confirm that the other server is not still issuing licenses. All new checkouts fail until the failed server is restored. Existing checkouts continue to work, but any job that finishes and releases its license cannot re-acquire it. This is why a triad is strongly recommended for environments where job startup latency is critical.

FlexNet Triad Failover: Master Failure and Recovery Compute Nodes 200 × Fluent jobs 4 tokens each lic-srv-01 Master (initial) ✗ Network partition at 10:00:00 lic-srv-02 Replica → Master ✓ Elected at 10:00:05 lic-srv-03 Replica ✓ Quorum maintained Timeline: Solver Job Startup Impact 10:00:00 Master fails 10:00:05 New master elected 10:00:35 Client retry succeeds 10:01:00 All jobs running Checkout failures (5s window) Client retry backoff (30s) Normal operation restored

Configuring Client Retry and Timeout Parameters

The client-side behavior during failover is controlled by environment variables and the license file. The most important parameters are:

  • FLEXLM_TIMEOUT — sets the connection timeout in microseconds. Default is 30,000,000 (30 seconds). Reducing this to 10 seconds can shorten the failover delay but may cause false timeouts on congested networks.
  • LM_RETRY — controls the number of retry attempts. Default is 2. Increasing this improves resilience but extends the worst-case startup delay.
  • FLEXLM_DIAGNOSTICS — set to 3 to enable verbose logging of license checkout attempts, which is invaluable for troubleshooting failover issues.

For a triad, the client library automatically tries the other servers if the master does not respond. However, this behavior depends on the USE_SERVER line and the order of SERVER lines in the license file. If the first server in the list is down, the client will attempt the second and third. This is why the order of servers in the license file should be consistent across all compute nodes.

Best practice: Set FLEXLM_TIMEOUT=10000000 (10 seconds) and LM_RETRY=3 on compute nodes in a triad environment. This balances fast failover with tolerance for transient network jitter. Always test failover by stopping lmgrd on one server and observing job startup behavior.

Monitoring and Verification

Verifying that a triad is healthy requires more than checking that lmgrd is running. Use lmstat -a -c 27000@lic-srv-01 to query the master. The output shows the status of all three servers, the number of licenses in use, and the vendor daemon status. A healthy triad shows all three servers as UP and the master as MASTER. If a server shows DOWN, the triad is running in degraded mode with two servers, which still provides quorum but no additional fault tolerance.

For automated monitoring, parse the lmstat output and alert if any server is down for more than 60 seconds. Also monitor the license usage trend: if the number of available tokens drops to zero during peak hours, jobs will queue even with a healthy triad. In that case, the solution is capacity expansion, not redundancy tuning.

Failure Symptoms and Troubleshooting

When a triad fails over, the most common symptom is a burst of Cannot connect to license server errors in solver logs. These errors are transient if the triad is healthy. If they persist, check the following:

  • Firewall rules: Ensure TCP ports 27000 and the vendor daemon port are open between all three servers and all compute nodes.
  • Hostname resolution: All servers must resolve each other's hostnames to the correct IP addresses. A mismatch causes quorum loss.
  • Clock skew: The three servers should have synchronized clocks (NTP). Large skew can cause election failures.
  • License file consistency: The license file must be identical on all three servers. A missing SERVER line causes the triad to fail at startup.

In a well-configured triad, a single server failure causes a 30–35 second delay for jobs that attempt a checkout during the election window. Jobs already running are unaffected. After the election, new checkouts succeed normally. This level of resilience is sufficient for most CAE production environments, where a 30-second delay is far preferable to a multi-hour outage.

Summary

High availability for CAE license servers is achieved through redundancy, and the triad configuration is the only FlexNet topology that provides true fault tolerance. A triad tolerates the failure of any one server by maintaining quorum with the remaining two. Failover is automatic and typically completes within 5 seconds, though client retry logic may add 30 seconds of startup delay for affected jobs. Proper configuration of client timeout and retry parameters, consistent license files, synchronized clocks, and open firewall ports are essential for reliable operation. Monitoring with lmstat and alerting on degraded states ensures that the license infrastructure remains a stable foundation for solver job startup across the HPC cluster.

← Previous 📖 Contents Next →