4. Submitting CAE jobs with sbatch, qsub, and bsub

Once a CAE model has been meshed and its solver input deck prepared, the next step is to hand that work to the cluster scheduler. On Linux HPC systems, three command-line interfaces dominate: sbatch (Slurm), qsub (PBS/Torque/OpenPBS/SGE), and bsub (LSF). They look similar on the surface — each takes a script and returns a job ID — but their option syntax, resource directives, and array semantics differ enough that a script written for one will not run unchanged on another. This section covers the syntax of each, the options that matter most for CAE workloads, and how to wrap solver invocations so that the same physics run can be launched across schedulers with minimal edits.

Common structure of a batch script

All three schedulers consume a shell script with two logical parts: a block of scheduler directives (interpreted by the scheduler, not the shell) and the executable body (the actual solver command). The directive block declares resources — nodes, cores, memory, wall time, queue/partition, output paths — and the body runs the solver. A minimal Slurm script for a structural analysis looks like this:

#!/bin/bash
#SBATCH --job-name=wing_static
#SBATCH --partition=compute
#SBATCH --nodes=2
#SBATCH --ntasks-per-node=32
#SBATCH --mem=128G
#SBATCH --time=04:00:00
#SBATCH --output=%x_%j.out
#SBATCH --error=%x_%j.err

module load abaqus/2023
abaqus job=wing_static input=wing.inp cpus=$SLURM_NTASKS scratch=$TMPDIR interactive

The equivalent PBS script uses #PBS directives and environment variables such as $PBS_NODEFILE:

#!/bin/bash
#PBS -N wing_static
#PBS -q compute
#PBS -l nodes=2:ppn=32
#PBS -l mem=128gb
#PBS -l walltime=04:00:00
#PBS -o wing_static.out
#PBS -e wing_static.err

cd $PBS_O_WORKDIR
module load abaqus/2023
abaqus job=wing_static input=wing.inp cpus=$(wc -l < $PBS_NODEFILE) interactive

And the LSF version uses #BSUB directives with -n, -R, and -W:

#!/bin/bash
#BSUB -J wing_static
#BSUB -q compute
#BSUB -n 64
#BSUB -R "span[ptile=32]"
#BSUB -M 131072
#BSUB -W 04:00
#BSUB -o wing_static.out
#BSUB -e wing_static.err

module load abaqus/2023
abaqus job=wing_static input=wing.inp cpus=$LSB_DJOB_NUMPROC interactive

Note the recurring pattern: each scheduler exposes the allocated core count through a different environment variable ($SLURM_NTASKS, $PBS_NODEFILE, $LSB_DJOB_NUMPROC). Hard-coding cpus=64 in the solver line is a common mistake — it desynchronizes the solver from the scheduler's actual allocation and either oversubscribes nodes or wastes cores.

Options that matter for CAE workloads

CAE jobs are typically long-running, memory-heavy, and often parameter-swept. The following options deserve attention across all three schedulers:

For CAE specifically, memory is the option most often mis-set. A CFD run with 40 million cells may need 200–400 GB per node; requesting too little causes the solver to be killed by the OOM killer mid-solve, while requesting too much delays scheduling. A practical rule is to measure peak RSS from a short pilot run and add 20–30% headroom.

Solver-specific wrappers

Most commercial CAE solvers ship with their own MPI launcher and expect to be invoked in a particular way. Rather than embedding solver-specific logic in every batch script, it is cleaner to write a thin wrapper that translates scheduler variables into solver arguments. A generic wrapper for Abaqus might look like:

#!/bin/bash
# abaqus_wrap.sh — usage: abaqus_wrap.sh <jobname> <input>
JOB=$1
INP=$2
NCORES=${SLURM_NTASKS:-${LSB_DJOB_NUMPROC:-$(wc -l < ${PBS_NODEFILE:-/dev/null})}}
module load abaqus/2023
abaqus job=$JOB input=$INP cpus=$NCORES mp_mode=mpi scratch=$TMPDIR interactive

Similar wrappers exist for ANSYS Mechanical APDL (ansys -dis -np $NCORES -b -i model.dat -o model.out), Fluent (fluent 3ddp -g -t$NCORES -i run.jou), and OpenFOAM (mpirun -np $NCORES simpleFoam -parallel). The wrapper pattern keeps the batch script scheduler-agnostic and centralizes solver-specific flags in one place.

Job arrays for parameter sweeps

CAE studies frequently require dozens or hundreds of runs — varying material properties, mesh densities, or boundary conditions. All three schedulers support job arrays, which submit one logical job that expands into many tasks. In Slurm:

#SBATCH --array=1-100
#SBATCH --output=run_%a.out

CASE=$(printf "case_%03d" $SLURM_ARRAY_TASK_ID)
abaqus job=$CASE input=${CASE}.inp cpus=$SLURM_NTASKS interactive

The PBS equivalent uses #PBS -t 1-100 with $PBS_ARRAYID, and LSF uses #BSUB -J "run[1-100]" with $LSB_JOBINDEX. Array jobs are efficient because the scheduler handles queueing and throttling; a common practice is to cap concurrency with --array=1-100%10 (Slurm) so that at most 10 tasks run simultaneously, preventing a sweep from monopolizing the cluster.

Dependencies and chained workflows

Many CAE workflows are multi-stage: mesh generation → solve → post-process → optimization loop. Scheduler dependencies let you express these as a DAG without polling. In Slurm, sbatch --dependency=afterok:$JOBID next.sh launches the next stage only if the first succeeded. PBS uses qsub -W depend=afterok:$JOBID, and LSF uses bsub -w "done($JOBID)". The afterok semantics are important: if the upstream job fails, the downstream job is not started, which prevents post-processing scripts from running on incomplete results.

Choosing the right submission path

If your site runs a single scheduler, the choice is made for you. In heterogeneous environments — for example, a university with a Slurm cluster and an LSF-based commercial cloud — the wrapper pattern shown above is the most maintainable approach: one solver wrapper, three thin batch scripts, and a small dispatch script that detects which scheduler is available (command -v sbatch, command -v qsub, command -v bsub) and submits accordingly. This keeps the physics and solver flags in one place while letting each site use its native scheduler.

The next step after submission is monitoring — tracking job state, detecting stalls, and reacting to failures — which builds directly on the job IDs and output files produced here.