3. Node-level health monitoring

In a CAE cluster, job failure rarely begins at the moment the solver aborts. Long before a CFD or FEA run crashes with an MPI communication error or an out-of-memory kill, the compute node has usually been drifting away from a healthy baseline: ECC correctable errors climbing, GPU memory temperature rising, CPU frequency throttling under sustained AVX-512 load, or a slow memory leak in a pre-processing daemon. Node-level health monitoring is the discipline of collecting the right metrics at the right cadence, exposing them through a standard interface, and turning them into signals that let you drain or repair a node before a 48-hour job dies at hour 40.

What to Measure on a CAE Compute Node

The metric set for a CAE node differs from a generic web server because the workload is characterized by sustained all-core floating-point throughput, large per-rank memory footprints, and (increasingly) GPU-accelerated solvers. The table below summarizes the categories that matter and the specific counters worth exporting.

Node Exporters: The Collection Layer

The de facto standard for Linux node metrics is node_exporter, which exposes a Prometheus-compatible /metrics endpoint on port 9100. For GPUs, dcgm-exporter (NVIDIA Data Center GPU Manager) exposes per-GPU metrics on port 9400. A minimal CAE node deployment looks like this:

# node_exporter with collectors relevant to CAE nodes
node_exporter \
  --collector.textfile.directory=/var/lib/node_exporter/textfile \
  --collector.cpu \
  --collector.meminfo \
  --collector.edac \
  --collector.hwmon \
  --collector.rapl \
  --collector.thermal_zone

# dcgm-exporter for GPU telemetry
dcgm-exporter --collectors /etc/dcgm-exporter/default-counters.csv

Two practical points are often overlooked. First, node_exporter does not read dmesg by default; kernel fault events must be scraped separately, for example by a small textfile collector script that greps for MCE, EDAC, or oom-kill and writes a gauge. Second, GPU ECC counters are only meaningful if you scrape them at a cadence shorter than the failure timescale — 15 s is a reasonable default, 60 s is too coarse for a node under heavy solver load.

Detecting Degradation Before Job Failure

Raw metrics are not alerts. The value of node-level monitoring comes from defining thresholds and trends that map to actionable states. A useful mental model is a three-tier escalation: watch, drain, and repair.

• Watch: Correctable ECC errors increasing, GPU temperature above 80 °C, CPU frequency below nominal for more than 5 minutes, swap usage above 1 % of RAM.
• Drain: Uncorrectable ECC error on any GPU, memory available below 5 % with active jobs, thermal zone above critical threshold, repeated MCE events.
• Repair: Node removed from scheduler, hardware ticket opened, node returned only after a burn-in test passes.

For CAE specifically, the most predictive single signal is often memory pressure trend. A solver that grows its resident set by 0.5 % per hour will not trigger an OOM alert until it is too late; a linear regression over node_memory_MemAvailable_bytes with a 6-hour window can project time-to-exhaustion and trigger a drain while the job can still be checkpointed. Similarly, a slow upward drift in GPU correctable ECC over days is a strong predictor of an uncorrectable failure during the next long run.

Putting It Together: A Node Health Signal Chain

The diagram below shows how metrics flow from hardware sensors through exporters into a time-series database, and how they are evaluated against thresholds to produce node states that the scheduler can act on.

From Alert to Action: A Worked Example

Consider a 64-core node running a 32-rank OpenFOAM case with 256 GB RAM. Over six hours, node_memory_MemAvailable_bytes drops from 180 GB to 40 GB, and swap-in rate rises from 0 to 12 MB/s. A naive threshold alert at 5 % available would not fire until 12.8 GB remained — by then the solver is already swapping heavily and performance has collapsed.

A better rule uses the derivative. With a 6-hour linear fit, the slope is approximately −23 GB/h. Projecting forward, the node reaches 10 % available (25.6 GB) in about 37 minutes. An alert of the form predict_linear(node_memory_MemAvailable_bytes[6h], 3600) < 0.1 * node_memory_MemTotal_bytes fires with roughly half an hour of headroom, enough to trigger a checkpoint and drain. This is the difference between a recoverable event and a lost 40-hour run.

For GPU nodes, the analogous signal is correctable ECC rate. A healthy H100 might show a handful of correctable errors per week; a degrading one shows hundreds per day. Exporting DCGM_FI_DEV_ECC_SBE_VOL_TOTAL and computing a rate over 24 hours gives a clean degradation indicator. When the rate exceeds a site-defined threshold (for example, 50 per hour sustained for 2 hours), the node should be drained and the GPU scheduled for diagnostics.

Operational Checklist

• Deploy node_exporter and dcgm-exporter on every compute node; verify /metrics responds and key series are present.
• Add a textfile collector script for kernel fault events (MCE, EDAC, oom-kill, soft lockup).
• Define recording rules for derived signals: memory availability percentage, ECC correctable rate, CPU frequency ratio, thermal headroom.
• Write alerting rules with for: clauses and predict_linear where trend matters more than level.
• Integrate node state with the scheduler so that a drain alert cordons the node and prevents new job placement.
• After repair, run a burn-in workload (e.g., stress-ng plus a short GPU benchmark) and confirm metrics return to baseline before returning the node to production.

Node-level health monitoring is not about collecting every possible metric. It is about selecting the counters that predict CAE job failure, exporting them reliably, and encoding the thresholds and trends that turn raw numbers into timely operational decisions. Done well, it converts unpredictable job losses into planned maintenance windows.