4. Installing PAM-CRASH on Linux clusters

A PAM-CRASH installation on a Linux cluster is not a single binary drop. It is a coordinated set of solver executables, shared libraries, MPI runtimes, license client components, and helper utilities that must agree on compiler ABI, MPI flavor, and library versions. Most failed deployments trace back to a mismatch in one of these three axes rather than to the solver itself. This section walks through the on-disk layout of a typical installation, how to expose it through environment modules, and how to resolve the dependency conflicts that appear when the cluster already hosts other CAE software.

Package layout and what each directory actually contains

PAM-CRASH distributions are usually delivered as a versioned tarball or as an RPM/DEB set. Regardless of packaging, the extracted tree follows a predictable structure. Understanding it matters because module files, license configuration, and MPI launch commands all reference specific subdirectories.

The critical detail is that lib/mpi/ is flavor-specific. A build compiled against Intel MPI will not run correctly under Open MPI even if the launcher name looks identical. The release notes in doc/ state which MPI versions were validated; treat that matrix as authoritative rather than assuming any recent MPI will work.

Environment modules: one module per variant, not one per version

On a shared cluster, the cleanest approach is to install each PAM-CRASH version once and expose it through a module that also loads the matching MPI and compiler runtime. The module should not silently pull in a system MPI that happens to be first in PATH.

# /opt/modulefiles/pamcrash/2023.0-intelmpi2021
#%Module1.0
proc ModulesHelp { } {
    puts stderr "PAM-CRASH 2023.0, Intel MPI 2021.9 build"
}

set root    /opt/pamcrash/2023.0-intelmpi2021
set version 2023.0

module-whatis "PAM-CRASH $version (Intel MPI 2021.9)"

prereq intelmpi/2021.9
prereq intel/2021.4

prepend-path PATH            $root/bin
prepend-path LD_LIBRARY_PATH $root/lib
prepend-path LD_LIBRARY_PATH $root/lib/mpi/intelmpi
prepend-path MANPATH         $root/doc/man

setenv PAM_CRASH_HOME   $root
setenv PAM_LICENSE_FILE 27000@license01.hpc.local

Two design choices in this file are deliberate. First, prereq rather than depends-on ensures the MPI module is already loaded and fails loudly if it is not, which prevents a silent fallback to a different MPI. Second, LD_LIBRARY_PATH is prepended, not appended, so the solver's own libraries take precedence over any identically named system libraries.

Dependency resolution in practice

Dependency problems on a CAE cluster usually fall into four categories: MPI ABI mismatch, missing or wrong-version shared libraries, license client incompatibility, and compiler runtime conflicts. The following diagram shows how a launch request flows through these layers and where each failure mode is detected.

Each failure point produces a distinct symptom. An MPI ABI mismatch typically aborts at MPI_Init with an undefined symbol referencing a versioned MPI function. A missing shared library is caught earlier by the dynamic loader, before the solver prints its banner. A license failure is the opposite: the solver starts, prints its version banner, then exits with a checkout error. Reading the point of failure tells you which layer to inspect.

Verifying the installation before handing it to users

A short verification sequence catches most problems before they reach production jobs. Run it once per build variant on a login node, then again inside a batch allocation.

# 1. Confirm the module resolves the intended paths
module load pamcrash/2023.0-intelmpi2021
which pamcrash_mpp
# expected: /opt/pamcrash/2023.0-intelmpi2021/bin/pamcrash_mpp

# 2. Check that all shared libraries resolve
ldd $(which pamcrash_mpp) | grep -i "not found"
# expected: no output

# 3. Confirm the MPI flavor matches the build
ldd $(which pamcrash_mpp) | grep -i mpi
# expected: libmpi.so from the Intel MPI module path, not /usr/lib64

# 4. Verify license reachability
lmutil lmstat -c 27000@license01.hpc.local -a | head -20
# expected: license server UP, PAM-CRASH feature listed

# 5. Run a small example deck under the batch scheduler
srun -N 2 -n 8 pamcrash_mpp -i examples/verification/small_deck.inp -o verify.out
# expected: normal termination, verify.out contains "NORMAL TERMINATION"

Step 3 is the one most often skipped. If ldd shows an MPI library from /usr/lib64 instead of the module path, the module's LD_LIBRARY_PATH is being overridden by a system-wide profile script. The fix is to ensure the module prepends rather than appends, and to check /etc/ld.so.conf.d/ for entries that inject a conflicting MPI path.

Handling multiple versions and variants on one cluster

Production clusters rarely host a single PAM-CRASH version. A workable convention is to install under /opt/pamcrash/<version>-<mpi> and create one module file per directory. Users then load the exact combination they need. To reduce confusion, set a default module alias that points to the currently validated production build, and document the supported matrix in a short table on the cluster wiki.

The dependency resolution work is largely front-loaded: once the module files correctly pin MPI and compiler runtimes, and once the verification sequence passes for each variant, users can select a build with a single module load and trust that the underlying libraries are consistent. The remaining operational task is to re-run the verification sequence whenever the cluster's system libraries, scheduler, or MPI modules are upgraded, since those changes can silently invalidate a previously working installation.