3. Python environment setup

A CAE automation script that runs flawlessly on your workstation can fail catastrophically on a colleague's machine or a CI runner because of a single mismatched library version. A solver wrapper pinned to numpy==1.24 may silently produce different interpolation results under numpy==1.26; a mesh-processing pipeline that depends on pyvista<0.44 may break when a transitive dependency upgrades vtk. Reproducibility in CAE automation is not a luxury — it is the difference between a result you can defend and a result you cannot. This chapter establishes the environment discipline that every subsequent automation project in this course will rely on.

Why Isolated Environments Are Non-Negotiable

Python installs packages into a shared site-packages directory by default. When two CAE projects require conflicting versions of the same library — say, Project A needs h5py 3.8 for reading solver output and Project B needs h5py 3.10 for a newer HDF5 feature — a single global environment cannot satisfy both. Virtual environments solve this by creating an isolated directory tree containing its own Python interpreter symlink and its own site-packages. Each project gets exactly the dependencies it declares, and nothing else.

For CAE work specifically, isolation also prevents accidental coupling between your automation toolchain and system-level packages. Many CAE platforms (Abaqus, ANSYS, STAR-CCM+) ship their own embedded Python interpreters. You should never install automation dependencies into the solver's embedded Python; instead, create a separate environment and call the solver through subprocess or its scripting API. This keeps the solver installation pristine and your automation environment independently upgradeable.

Creating and Managing Virtual Environments

The standard-library approach uses venv. From your project root:

python -m venv .venv
# Linux / macOS
source .venv/bin/activate
# Windows (PowerShell)
.venv\Scripts\Activate.ps1

Once activated, python and pip resolve to the environment's copies. Verify with which python (Linux/macOS) or where python (Windows) — the path must point inside .venv. Deactivate with deactivate when switching contexts.

For CAE projects that involve heavy scientific stacks, conda (or its faster sibling mamba) is often preferable because it manages non-Python binary dependencies — BLAS/LAPACK, MPI, HDF5, NetCDF — that pip cannot reliably provision. A typical conda environment for a CFD post-processing project:

conda create -n cfd-post python=3.11
conda activate cfd-post
conda install numpy scipy h5py matplotlib pyvista

The choice between venv and conda is pragmatic: use venv when your dependencies are pure-Python or have reliable wheels on PyPI; use conda when you need compiled scientific libraries with specific build configurations (e.g., MPI-linked mpi4py matching your cluster's MPI implementation).

Dependency Management: From requirements.txt to Lock Files

A requirements.txt file lists direct dependencies, but it does not pin transitive dependencies. If your project depends on pyvista, which depends on vtk, which depends on numpy, a requirements.txt containing only pyvista will install whatever versions pip resolves at install time — which may differ tomorrow. For reproducible CAE automation, you need a lock file that records the exact version of every package in the dependency graph.

Two modern tools dominate this space: pip-tools and Poetry. With pip-tools, you maintain a requirements.in with loose constraints and compile a fully pinned requirements.txt:

# requirements.in
numpy>=1.24,<2.0
scipy>=1.11
h5py>=3.8
pyvista>=0.43

# Compile
pip-compile requirements.in --output-file requirements.txt

# Install exactly what was locked
pip-sync requirements.txt

The generated requirements.txt contains lines like numpy==1.26.4 and vtk==9.3.0, each with a hash. Committing this file to version control guarantees that every team member and every CI run installs the identical set. When you need to upgrade, you edit requirements.in and re-compile — never hand-edit the lock file.

Poetry takes a more integrated approach, managing both dependency resolution and packaging in a single pyproject.toml:

[tool.poetry]
name = "cae-automation"
version = "0.1.0"
description = "Batch execution and monitoring for CAE solvers"

[tool.poetry.dependencies]
python = "^3.11"
numpy = ">=1.24,<2.0"
scipy = "^1.11"
h5py = "^3.8"
pyvista = "^0.43"

[tool.poetry.group.dev.dependencies]
pytest = "^8.0"
ruff = "^0.4"

Running poetry lock produces a poetry.lock file with exact versions and hashes. poetry install then recreates the environment deterministically. Poetry also separates runtime dependencies from development dependencies, which keeps production environments lean — important when deploying automation scripts to a compute cluster where you do not want pytest and ruff installed.

Packaging Your Automation Project

Once your environment is reproducible, the next step is making your automation code itself installable and importable. A well-structured CAE automation package follows the src-layout convention:

cae-automation/
├── pyproject.toml
├── poetry.lock
├── src/
│   └── cae_automation/
│       ├── __init__.py
│       ├── solver_runner.py
│       ├── monitor.py
│       └── postprocess.py
└── tests/
    └── test_solver_runner.py

The pyproject.toml declares the package metadata and entry points. An entry point lets you expose a CLI command directly from your package:

[tool.poetry.scripts]
cae-run = "cae_automation.solver_runner:main"

After poetry install, typing cae-run --help invokes your main() function. This is how you turn a collection of scripts into a tool that colleagues can install with a single command and invoke without knowing the internal module structure.

Environment Reproducibility in Practice

The following diagram illustrates the complete environment lifecycle for a CAE automation project, from declaration through locked installation to execution on a compute node.

Verifying Reproducibility

After setting up the environment, verify it before trusting any automation results. A minimal verification script checks that the installed versions match the lock file and that a known computation produces the expected output:

# verify_env.py
import numpy as np
import h5py
import sys

print(f"Python: {sys.version}")
print(f"numpy:  {np.__version__}")
print(f"h5py:   {h5py.__version__}")

# Deterministic check: same seed → same array
rng = np.random.default_rng(seed=42)
sample = rng.random(5)
expected = np.array([0.77395605, 0.43887844, 0.85859792, 0.69736803, 0.09417735])
assert np.allclose(sample, expected, atol=1e-7), "RNG output mismatch — environment may differ"
print("Environment verification passed.")

Run this on every machine that will execute your automation. If the assertion fails, the environment is not reproducible — check that the lock file was used, that no packages were installed manually, and that the Python minor version matches.

Practical Checklist for CAE Automation Projects

With a locked, verified environment in place, your CAE automation scripts become portable artifacts: the same code, the same dependencies, and the same numerical results regardless of whether they run on a laptop, a departmental workstation, or a cluster login node. The next chapter builds on this foundation by introducing the batch execution patterns that drive solver runs from within this controlled environment.