2. Linux job scheduling landscape

On a CAE cluster, the scheduler is the single component that decides which simulation runs, where it runs, and when it starts. A solver such as Abaqus, ANSYS Fluent, or OpenFOAM may be perfectly configured, but without a batch scheduler the cluster degenerates into a set of machines that engineers log into manually. The four schedulers you will encounter most often in CAE environments are SLURM, PBS Pro, LSF, and SGE (Son of Grid Engine, also known as Grid Engine). They differ in syntax and internal architecture, but they share the same conceptual model: users describe a job's resource requirements, the scheduler places that job on suitable nodes, and a daemon on each node executes and reports on it.

The Common Conceptual Model

Every one of these systems implements the same four abstractions, even though the names change:

• Job — a script plus a resource request (CPU cores, memory, wall-clock time, GPUs, licenses).
• Queue / Partition — a logical pool of nodes with shared limits and priorities. SLURM calls it a partition; PBS Pro, LSF, and SGE call it a queue.
• Resource request — the declaration of what the job needs. This is the single most important thing a CAE engineer writes, because an under-specified request leads to either wasted nodes or out-of-memory kills.
• Job states — pending, running, suspended, completed, failed. Monitoring automation almost always keys off these states.

The terminology mapping below is worth memorizing, because CAE vendor documentation and site runbooks mix these terms freely.

Architecture: Controller, Scheduler, and Node Daemons

All four systems follow a controller/executor split. A central manager daemon holds the authoritative state of every job and node. A scheduler daemon (sometimes a plugin inside the manager) makes placement decisions. On each compute node, an execution daemon receives the job, sets up the environment, launches the solver, and streams status back. The diagram below shows this topology with the real daemon names for each system.

What Each Scheduler Does Differently in Practice

SLURM is the dominant choice for new CAE clusters. Its resource model is task-based: --ntasks sets the number of MPI ranks, and --cpus-per-task sets threads per rank. This maps cleanly onto hybrid MPI+OpenMP solvers. SLURM also exposes sacct, which returns structured accounting records — the foundation for any monitoring automation you will build later in this course.

PBS Pro (now Altair PBS Professional) uses a chunk-based request syntax: -l select=2:ncpus=16:mpiprocs=16:mem=64gb. This is more verbose but expresses heterogeneous requests naturally, which matters when a job needs, say, two 16-core nodes plus one 8-core node for a pre-processing step. PBS Pro is common in commercial CAE shops because of its mature license-aware scheduling.

LSF (IBM Spectrum LSF) is built around resource requirement strings and rusage reservations. A typical CAE submission looks like bsub -n 32 -R "rusage[mem=4096]" -W 24:00 -q normal fluent -t4 -i case.jou. LSF's strength is its fair-share and preemption policy engine, which many large aerospace and automotive sites rely on.

SGE is the oldest of the four and is still found on legacy clusters. Its distinguishing feature is the parallel environment (-pe), which defines how a job's slots are distributed across nodes. SGE's qstat -f output is notoriously verbose, which is exactly why monitoring automation for SGE tends to parse it with structured filters rather than raw text.

A Concrete Comparison: Submitting the Same Fluent Job

Suppose you need to run a Fluent case on 32 cores across two nodes, with 4 GB of memory per core and a 24-hour wall-clock limit. The four submission scripts differ only in their directive syntax:

# SLURM
#SBATCH --job-name=fluent_case
#SBATCH --nodes=2
#SBATCH --ntasks=32
#SBATCH --cpus-per-task=1
#SBATCH --mem-per-cpu=4G
#SBATCH --time=24:00:00
#SBATCH --partition=normal
srun fluent 3ddp -t32 -i case.jou

# PBS Pro
#PBS -N fluent_case
#PBS -l select=2:ncpus=16:mpiprocs=16:mem=64gb
#PBS -l walltime=24:00:00
#PBS -q normal
mpirun -np 32 fluent 3ddp -t32 -i case.jou

# LSF
#BSUB -J fluent_case
#BSUB -n 32
#BSUB -R "rusage[mem=4096]"
#BSUB -W 24:00
#BSUB -q normal
mpirun -np 32 fluent 3ddp -t32 -i case.jou

# SGE
#$ -N fluent_case
#$ -pe mpi 32
#$ -l h_rt=24:00:00
#$ -l h_vmem=4G
#$ -q normal
mpirun -np 32 fluent 3ddp -t32 -i case.jou

Notice that the resource semantics are identical, but the syntax is not. This is why a monitoring automation layer should never hard-code a single scheduler's command syntax. Instead, it should abstract over a small set of operations — submit, query state, cancel, retrieve exit code — and implement one adapter per scheduler.

Why This Matters for Automation

The rest of this course builds monitoring and automation on top of these schedulers. Before you can poll job status, detect a stalled solver, or auto-resubmit a failed case, you need to know which daemon owns the state, which command queries it, and what the returned state string actually means. The architecture diagram above is the mental model; the terminology table is the translation layer; and the four submission scripts are the concrete artifacts you will adapt. In the next sections, we will use SLURM as the primary reference implementation because of its clean accounting interface, but every automation pattern will be presented with the corresponding PBS Pro, LSF, and SGE equivalents so that the techniques transfer to whichever scheduler your site runs.