2. Python environment setup for CAE data processing

CAE post-processing scripts rarely fail because the extraction logic is wrong. More often they fail because the interpreter that runs the script is not the interpreter that has pandas installed, because a solver output directory is resolved relative to the wrong working directory, or because a colleague's machine silently upgraded numpy and broke binary compatibility with a compiled reader. This chapter treats the Python environment as an engineering deliverable: a reproducible, path-aware, dependency-pinned runtime that can be handed to another analyst and behave identically.

Choosing the interpreter and isolating the project

For CAE data processing, use a 64-bit CPython build (3.10–3.12 is a safe current range) and never install analysis libraries into the system interpreter. Create a dedicated virtual environment per project so that a solver-vendor SDK, a specific numpy ABI, and an Excel writer library can be pinned together without contaminating other work.

# Windows (PowerShell), from the project root
py -3.11 -m venv .venv
.\.venv\Scripts\Activate.ps1

# Linux / WSL
python3.11 -m venv .venv
source .venv/bin/activate

# Confirm you are inside the environment
python -c "import sys; print(sys.executable)"

The printed path must point inside .venv. If it points to C:\Python311\python.exe or /usr/bin/python3, activation failed and every subsequent pip install will land in the wrong place. This single check eliminates the most common class of "module not found" reports in CAE scripting teams.

The library set for injury-metric extraction and Excel reporting

The dependency set is small but each package has a specific role. Install them explicitly rather than pulling a generic "data science" bundle, so the lockfile stays auditable.

python -m pip install --upgrade pip
python -m pip install numpy==1.26.4 pandas==2.2.2 scipy==1.13.1 \
    openpyxl==3.1.2 pyyaml==6.0.1 pytest==8.2.0

Pin exact versions in requirements.txt and commit it. When a compiled dependency such as scipy is involved, exact pins prevent the wheel-resolution drift that produces different filter results between machines. For a fully reproducible lock including transitive dependencies, generate one with pip freeze > requirements.lock and install from the lock in CI or on a colleague's workstation.

Configuring paths without hard-coding them

CAE projects move. A run directory that is D:\crash\run_042 on the analyst's workstation becomes /mnt/projects/crash/run_042 on the Linux cluster. Hard-coded absolute paths are the second most common source of script failure. Resolve every path from a single configuration root, and keep the resolution logic in one module.

# config.py
from pathlib import Path
import os, yaml

PROJECT_ROOT = Path(os.environ.get("CAE_PROJECT_ROOT", Path(__file__).parent))
CONFIG_PATH  = PROJECT_ROOT / "config" / "run_config.yaml"

def load_config():
    with open(CONFIG_PATH, "r", encoding="utf-8") as fh:
        cfg = yaml.safe_load(fh)
    # Resolve every declared path against the project root
    for key in ("run_dir", "channel_map", "report_dir"):
        cfg[key] = (PROJECT_ROOT / cfg[key]).resolve()
    return cfg

With CAE_PROJECT_ROOT set per machine, the same code runs locally and on the cluster. The resolve() call normalizes separators and collapses .. segments, which matters when a config file uses relative paths like ../runs/run_042. Validate existence early and fail loudly:

cfg = load_config()
for key in ("run_dir", "channel_map"):
    if not cfg[key].exists():
        raise FileNotFoundError(f"{key} not found: {cfg[key]}")

For solver output, also confirm the expected file pattern is present (for example *.binout, *.d3plot, or *.csv) before parsing. A directory that exists but contains a truncated or zero-byte output is a different failure than a missing directory, and the error message should distinguish them.

How the pieces connect at runtime

The diagram below shows the runtime resolution chain: the activated interpreter selects the pinned library set, the environment variable anchors the project root, and the configuration file maps logical names to concrete solver output and report locations. Reading it left to right reproduces the order in which the script resolves each dependency.

Verifying the environment before trusting it

Do not wait for a full extraction run to discover a broken environment. A short smoke test that imports each library, prints versions, and exercises one path resolution catches nearly all setup defects in seconds.

# smoke_test.py
import sys, numpy, pandas, scipy, openpyxl, yaml
from config import load_config

print("python :", sys.version.split()[0])
print("numpy  :", numpy.__version__)
print("pandas :", pandas.__version__)
print("scipy  :", scipy.__version__)
print("openpyxl:", openpyxl.__version__)

cfg = load_config()
print("run_dir   :", cfg["run_dir"], cfg["run_dir"].exists())
print("report_dir:", cfg["report_dir"])

Expected output is a version line per library and True for the run directory. Typical failure symptoms and their causes:

• ModuleNotFoundError: No module named 'pandas' — the environment is not activated, or pip installed into a different interpreter. Re-check sys.executable.
• ImportError: DLL load failed on Windows — a 32-bit interpreter or a mismatched numpy/scipy wheel. Reinstall both from the same index with matching versions.
• FileNotFoundError: run_dir not found — CAE_PROJECT_ROOT points to the wrong level, or the config uses a relative path that resolves outside the project. Print the resolved path, not the raw string.
• Report opens but formulas show #REF! — the workbook was written by a library version that changed formula handling; pin openpyxl and re-run.

Dependency hygiene across a team

Treat requirements.txt as source code: review changes, pin exact versions, and regenerate the lock whenever a package is added. When a new library is genuinely required, add it in a branch, run the smoke test and the extraction regression tests, and only then merge. For air-gapped engineering networks, pre-download wheels with pip download -r requirements.txt -d wheels/ and install offline via pip install --no-index --find-links wheels/ -r requirements.txt. This keeps the CAE analysis environment reproducible regardless of network policy, and ensures that the numbers in the Excel report depend on the model, not on which machine happened to run the script.