2. Monitoring fundamentals for CAE clusters

CAE clusters are not general-purpose web fleets. A single explicit dynamics job may pin hundreds of cores for hours, saturate the InfiniBand fabric with MPI traffic, and write multi-gigabyte result files to a parallel filesystem. Monitoring such an environment requires understanding four distinct telemetry types — metrics, logs, traces, and events — and how they are modeled as time-series data. This section establishes that foundation before we move to specific tooling.

The Four Telemetry Signals

Metrics are numeric measurements sampled at regular intervals: CPU utilization, memory bandwidth, fabric port counters, Lustre OST throughput, job queue depth. They are cheap to store and ideal for dashboards and alerting, but they lose context — a CPU spike tells you what happened, not why.

Logs are discrete, timestamped text records emitted by the scheduler (Slurm slurmctld), the resource manager, MPI libraries, and node-level daemons. They carry rich context but are expensive to index at scale and are unstructured unless you enforce a format.

Traces capture the causal path of a request or job across components. In HPC, distributed tracing is less mature than in microservices, but MPI profiling tools (e.g., mpiP, Score-P) and job-level tracing through the scheduler provide span-like data showing where time is spent across ranks and phases.

Events are state transitions: a job entering RUNNING, a node transitioning to DRAIN, a filesystem mounting, a GPU falling off the bus. Events are sparse but high-value — they explain when something changed, which is often the missing link between a metric anomaly and a log entry.

Time-Series Data Model

Metrics and events are stored as time-series data. The canonical model is a measurement (e.g., node_cpu_seconds_total) with a set of tags (labels) that identify the series — hostname, cpu, mode — and a field holding the value at a timestamp. This is the InfluxDB/Prometheus lineage. The key design decision is cardinality: every unique tag combination creates a new series. Tagging by jobid on a cluster running 50,000 jobs per day will explode your index. Tag by partition or node instead, and keep jobid in logs or events.

For CAE workloads, the most valuable time-series are:

• Node-level: CPU utilization per core, memory used, load average, GPU utilization and memory (if applicable), NIC and HCA counters.
• Fabric-level: InfiniBand port XmitWait, PortRcvErrors, LinkDowned; OmniPath/RoCE equivalents.
• Storage-level: Lustre/GPFS client and server throughput, metadata operation latency, OST utilization.
• Scheduler-level: pending/running job counts, queue wait time, node state distribution.

Observability Principles in HPC Context

Observability is not just "having dashboards." It is the ability to ask arbitrary questions about system state without pre-defining every possible failure mode. Three principles matter most in CAE clusters:

1. Correlate across signals. A job slowdown might show as low CPU utilization (metric), a DRAIN event on a node, and an MPI retry storm in logs. The monitoring system must let you pivot from one to the other by time window and node/job identifier.

2. Respect the sampling and retention trade-off. High-frequency sampling (1s) is needed for bursty HPC workloads but expensive. A tiered strategy — 1s for 24h, 1m for 30d, 1h for 1y — balances resolution against cost. Downsampling must preserve peaks, not just averages, or you will miss transient saturation.

3. Monitor the monitoring system. If your metrics collector falls behind, you get stale data that looks like a healthy cluster. Track scrape duration, queue depth, and write latency of the monitoring stack itself.

Applying the Model: A Concrete Example

Consider a user reporting that a 256-core OpenFOAM job ran 40% slower than last week. A structured investigation uses all four signals:

1. Metrics: Query node CPU utilization for the job's nodes over the run window. If utilization is high but throughput is low, suspect memory bandwidth or fabric contention.
2. Events: Check for node state changes or maintenance events during the window. A single DRAIN event on one node can force MPI ranks to wait.
3. Logs: Grep the job's stderr for MPI warnings, slurmstepd messages, or filesystem timeouts.
4. Traces: If available, compare MPI communication time versus computation time between the slow run and a baseline.

The following SVG illustrates how these four signals flow from cluster components into a unified observability layer, and how a single job-level question is answered by correlating them.

Practical Guidance for CAE Cluster Monitoring

When designing or evaluating a monitoring stack for CAE workloads, keep these points in mind:

• Sample at the right frequency. CPU and memory at 10–15s is usually sufficient; fabric error counters and job state changes should be event-driven or sampled at 1s during active jobs.
• Use consistent labels. Adopt a naming convention across all exporters: cluster, partition, hostname, jobid (only in logs/events), user (careful with cardinality).
• Retain raw data long enough to debug. A job that ran last Tuesday may be reported as slow today. Keep at least 30 days of 1-minute resolution data.
• Alert on symptoms, not causes. Alert when job queue wait time exceeds a threshold or when node utilization drops below expected for a running job — not when a single counter crosses an arbitrary line.

With these fundamentals in place — the four signal types, the time-series model, and the correlation principle — the next section examines how to instrument a CAE cluster end-to-end, from node exporters to scheduler hooks, and how to validate that the telemetry pipeline itself is healthy.