4. Job scheduler and workload monitoring

On a CAE cluster, the scheduler is the component that decides which simulation job runs where and when. Monitoring it is not the same as monitoring nodes. A node can be perfectly healthy while the scheduler is congested, misconfigured, or silently rejecting work. This section covers the four schedulers most commonly found in CAE environments — Slurm, PBS (PBS Professional / OpenPBS), LSF, and Kubernetes — and how to read their job states, queue depth, pending reasons, and health indicators.

Job state models across schedulers

Every scheduler exposes a state machine, but the vocabulary differs. The practical skill is mapping each scheduler's states onto a common mental model: accepted → waiting → running → terminal. Terminal states split into success, failure, and cancellation.

For Kubernetes, a CAE job is usually wrapped in a Job or Pod. The pod phase (Pending, Running, Succeeded, Failed, Unknown) is the closest analogue, but the real diagnostic detail lives in status.conditions and container waiting.reason fields such as ImagePullBackOff or CrashLoopBackOff.

Queue depth and pending reasons

Queue depth is the count of jobs in a waiting state, ideally broken down by partition/queue and by user or project. A rising queue depth is not automatically a problem — it may simply reflect a large batch submission. The actionable signal is why jobs are pending. Slurm exposes this directly:

$ squeue -h -t PENDING -o "%i %r" | sort | uniq -c | sort -rn
     42 Resources
     17 Priority
      6 QOSMaxJobsPerUserLimit
      2 Dependency

Each reason maps to a different remedy:

• Resources — no node currently satisfies the request (cores, memory, GPUs, or a specific feature). Check whether the request is realistic against partition limits.
• Priority — higher-priority jobs are ahead. This is normal; investigate only if it persists beyond expected turnaround.
• QOSMaxJobsPerUserLimit — a policy cap is throttling one user. Raise the limit or ask the user to consolidate jobs.
• Dependency — the job waits on another job. Verify the dependency chain is not deadlocked.

PBS reports pending reasons via qstat -f in the comment attribute, and LSF via bjobs -p. Kubernetes surfaces scheduling failures in the pod's status.conditions with reason Unschedulable and a message such as 0/12 nodes are available: 12 Insufficient nvidia.com/gpu. The pattern is identical across all four: pending reason → resource or policy constraint → targeted fix.

Scheduler health indicators

Beyond individual jobs, the scheduler itself has health signals that predict outages before they happen. The most important are:

• Controller responsiveness — Slurm's slurmctld, PBS's pbs_server, LSF's mbatchd, and the Kubernetes kube-scheduler should all answer status queries within a bounded time. A growing response latency is an early warning.
• Backlog age — the age of the oldest pending job. If it exceeds the expected maximum queue time, capacity or policy is the bottleneck.
• Throughput — jobs started and completed per unit time. A drop in throughput with stable queue depth usually means jobs are running longer, not that the scheduler is failing.
• Node registration — the count of nodes known to the scheduler versus the count expected. Missing nodes indicate daemon or network problems.
• Database / state health — Slurm's slurmdbd, PBS's datastore, and LSF's mbatchd state files must be writable and consistent. Corruption here causes silent job loss.

Reading the scheduler as a system

The diagram below shows how a job flows through a generic scheduler and where each monitoring signal is observed. The same structure applies whether the scheduler is Slurm, PBS, LSF, or Kubernetes — only the command names and state labels change.

Practical monitoring workflow

A reproducible check for a Slurm cluster, run from a login node, looks like this:

# 1. Overall queue depth by state
squeue -h -o "%t" | sort | uniq -c

# 2. Pending reasons ranked
squeue -h -t PENDING -o "%r" | sort | uniq -c | sort -rn

# 3. Oldest pending job age
squeue -t PENDING -o "%i %V" --sort=V | head -5

# 4. Node availability
sinfo -s

# 5. Controller health
scontrol ping

Expected output for a healthy cluster: pending count is small relative to running count, the dominant pending reason is Priority (normal contention), the oldest pending job is within the expected turnaround window, sinfo shows all nodes in idle or alloc, and scontrol ping returns UP for both primary and backup controllers.

Typical failure symptoms and their interpretation:

• scontrol ping returns DOWN — the controller is unreachable. Check the slurmctld service and its logs before touching jobs.
• Pending reason is overwhelmingly Resources with idle nodes present — a feature, GPU, or memory request cannot be satisfied. Compare the job's --gres and --mem against sinfo -o "%N %G %m".
• Queue depth is flat but throughput drops — jobs are running longer. Inspect per-job elapsed time, not the scheduler.
• Nodes disappear from sinfo — the slurmd daemon or the node's network path is failing. Check slurmd logs on the affected node.

For PBS, the equivalent checks are qstat -q for queue limits, qstat -f for per-job comment fields, and pbsnodes -a for node state. For LSF, use bqueues, bjobs -p, and lsload. For Kubernetes, kubectl get pods --field-selector=status.phase=Pending followed by kubectl describe pod <name> reveals the scheduling condition and message.

What to record and alert on

A minimal, useful monitoring set for any of the four schedulers includes: queue depth per partition, pending reason distribution, oldest pending job age, running job count, node state summary, and controller reachability. These six signals, sampled every one to five minutes, are enough to distinguish normal contention from a genuine scheduler or capacity problem. Alert thresholds should be derived from the cluster's own historical turnaround time, not from generic defaults.