2. Multi-team HPC platform requirements

An automotive OEM rarely runs CAE workloads for a single team. A typical platform serves crashworthiness, NVH, CFD, thermal management, durability, and powertrain groups, plus external suppliers working on body-in-white, battery packs, or ADAS sensor housings. Each of these groups has different confidentiality obligations, different software stacks, and different notions of what "fair" means when the cluster is full. The platform requirements that follow from this reality are not generic HPC concerns — they are the constraints that determine whether the platform is usable at all in a multi-organization setting.

Tenant Isolation

A tenant is any logically separated consumer of platform resources: an internal department, a supplier, a joint venture, or a program-specific team. Isolation must be enforced at several layers simultaneously, because a gap at any one layer can leak data or destabilize neighbors.

At the identity layer, each tenant maps to a distinct identity provider realm or at minimum a distinct group hierarchy in LDAP/AD. Job submission credentials should be short-lived and scoped to the tenant. At the scheduler layer, tenants are represented as accounts or projects with their own quotas, priorities, and preemption policies. At the execution layer, jobs from different tenants should not share nodes unless the platform explicitly supports gang-scheduled co-tenancy with cgroup-enforced CPU, memory, and GPU limits. At the network layer, InfiniBand partition keys (P_Keys) or equivalent fabric segmentation prevent cross-tenant MPI traffic. At the storage layer, each tenant gets a separate filesystem namespace or at least a separate export with POSIX ACLs and root_squash enforced.

A common failure mode is partial isolation: the scheduler separates tenants but the shared scratch filesystem does not. A supplier's solver job can then read another supplier's mesh files simply by guessing the path. The requirement is not "isolation somewhere" but "isolation at every layer that carries tenant data or tenant traffic."

Fair Share Scheduling

Fair share in a multi-tenant CAE platform is not the same as equal share. Crash teams running 24-hour LS-DYNA jobs and CFD teams running 8-hour Fluent jobs have different queueing behavior, and a naive equal-share policy will starve one or the other. The scheduler must support hierarchical fair share with configurable weights, decay half-lives, and preemption rules.

Slurm's FairShare and PriorityWeightFairshare parameters, or PBS Pro's fair_share_percentage, allow the platform team to assign each tenant a target share of the cluster. The key parameters to define per tenant are:

• Share weight — relative entitlement, e.g., crash=40, NVH=20, CFD=20, suppliers=20.
• Decay half-life — how quickly historical usage is forgotten. A 7-day half-life suits project-based CAE; a 30-day half-life suits steady-state production.
• Grace period — the usage window before fair share penalties apply, preventing a tenant from being penalized for a single large job.
• Preemption policy — whether high-priority tenants can preempt low-priority jobs, and whether preempted jobs are requeued or killed.

For supplier tenants, fair share must also account for the fact that their jobs may be bursty and deadline-driven. A supplier delivering a crash model for a regulatory submission may need a temporary priority boost, which the platform should support through a time-bounded reservation or a QoS tier rather than a permanent weight change.

Data Segregation

Data segregation in automotive CAE is driven by three forces: intellectual property boundaries between OEM and supplier, export control regulations (ITAR, EAR) that restrict access to certain geometry or material data, and contractual confidentiality clauses that may prohibit co-location of data from competing suppliers on the same physical storage.

The practical implementation is a per-tenant storage layout with explicit access control. A workable pattern is:

/fs/cae/
  tenants/
    oem_crash/        # owner: oem-crash, mode 2770
    oem_nvh/          # owner: oem-nvh, mode 2770
    supplier_a/       # owner: sup-a, mode 2770
    supplier_b/       # owner: sup-b, mode 2770
  shared/
    reference_models/ # read-only for all tenants
    solver_binaries/  # read-only for all tenants

Each tenant directory is owned by a tenant-specific group, with setgid set so new files inherit the group. Cross-tenant reads are denied by default. Shared reference data is mounted read-only. If export-controlled data must be processed, it should live on a separate filesystem with additional access logging and, where required, encryption at rest with tenant-specific keys.

Data segregation also applies to job output. Solver logs, result files, and post-processing artifacts must land in the tenant's own namespace. A shared scratch area is acceptable only if it is ephemeral, purged on a short schedule, and not readable across tenants.

Compliance Drivers

Compliance requirements shape the platform more than any other factor, because they are externally audited and non-negotiable. The main drivers in automotive OEM and supplier environments are:

The platform must produce audit evidence: who submitted which job, which tenant it belonged to, what data it accessed, and where the output was written. This is not optional logging — it is a design requirement that affects the scheduler, the filesystem, and the identity provider simultaneously.

Translating Requirements into Platform Configuration

The requirements above become concrete only when they are expressed as configuration. A minimal Slurm-based implementation for a four-tenant platform might look like this:

# slurm.conf — account and QoS definitions
AccountingStorageEnforce=associations,qos,limits
PriorityType=priority/multifactor
PriorityWeightFairshare=10000
PriorityWeightQOS=5000
PriorityDecayHalfLife=7-0
PriorityUsageResetPeriod=MONTHLY

# sacctmgr — tenant accounts
sacctmgr add account oem_crash Description="OEM Crash" Fairshare=40
sacctmgr add account oem_nvh   Description="OEM NVH"   Fairshare=20
sacctmgr add account sup_a     Description="Supplier A" Fairshare=20
sacctmgr add account sup_b     Description="Supplier B" Fairshare=20

# QoS for deadline-driven supplier work
sacctmgr add qos supplier_deadline Priority=1000 Preempt=normal GraceTime=3600

On the storage side, the corresponding enforcement is a combination of POSIX permissions, setgid directories, and, where export control applies, a separate mount with noexec,nosuid and audit logging via auditd rules on the tenant path. The scheduler and the filesystem must agree on tenant identity — if the scheduler knows a job belongs to sup_a but the filesystem sees only a shared service account, segregation is lost.

Verification and Failure Symptoms

Each requirement should be verified with a concrete test. For tenant isolation, submit a job as sup_a that attempts to read /fs/cae/tenants/sup_b/ and confirm permission denied. For fair share, run synthetic load from two tenants with different weights and confirm that the higher-weight tenant receives proportionally more node-hours over a 24-hour window. For data segregation, verify that a job's output lands only in the submitting tenant's namespace and that no cross-tenant symlinks exist. For compliance, confirm that sacct records the account, user, and job ID for every submission and that the audit log captures file access on export-controlled paths.

Typical failure symptoms include: jobs from one tenant appearing in another tenant's squeue output (identity mapping error), fair share not converging because PriorityDecayHalfLife is too long relative to job duration, solver jobs writing to a shared scratch path because the working directory was not set per tenant, and audit gaps because the scheduler and filesystem use different identity sources. Each of these points back to a specific layer in the diagram above, and each is fixable with a targeted configuration change rather than a platform redesign.