4. Subprocess fundamentals

Subprocess fundamentals 学习卡片

Every batch execution system for a CAE solver ultimately reduces to one operation: starting an external process, feeding it the right arguments and environment, waiting for it to finish, and inspecting what it produced. Python's subprocess module is the standard tool for this job. This section covers the three entry points you will use most — subprocess.run, subprocess.call, and subprocess.Popen — and the three concerns that cause the most production failures: argument construction, environment variables, and working directories.

Choosing the Right Entry Point

The three functions are not interchangeable; they sit at different levels of control.

APIBlockingReturnsTypical use
subprocess.callYesExit code onlyFire-and-forget checks where you only need success/failure
subprocess.runYesCompletedProcess (exit code, stdout, stderr)Default choice for solver runs with captured output
subprocess.PopenNoProcess handleConcurrent solves, streaming logs, timeouts, killing jobs

For a single solver invocation where you want the log text back, run is the right default. Reach for Popen only when you need to do something while the solver runs — poll progress, stream the log, enforce a wall-clock limit, or launch several jobs at once.

Building the Argument List

CAE solvers are driven almost entirely by command-line arguments. A typical Abaqus job looks like this on the command line:

abaqus job=crash_01 input=crash_01.inp cpus=4 interactive

In Python, pass this as a list, never as a single string with shell=True:

import subprocess

cmd = [
    "abaqus",
    "job=crash_01",
    "input=crash_01.inp",
    "cpus=4",
    "interactive",
]
result = subprocess.run(cmd, capture_output=True, text=True)
print(result.returncode)
print(result.stdout[-2000:])

Each list element becomes one argv entry. This matters because solver arguments frequently contain =, spaces in paths, or characters the shell would interpret. With a list and no shell, no quoting or escaping is needed — the operating system receives the arguments exactly as written. Using shell=True with an interpolated string is the single most common source of broken batch scripts: a path like C:\Users\John Doe\model.inp splits at the space, and a filename containing & or ; can execute unintended commands.

Never build solver commands by string concatenation. If a job name or path comes from user input or a file listing, string interpolation plus shell=True is a command-injection risk. Always pass a list.

Environment Variables

CAE solvers depend heavily on environment variables: license server addresses (LM_LICENSE_FILE, ABAQUSLM_LICENSE_FILE), MPI configuration (I_MPI_ROOT, PATH), scratch directories (TMPDIR), and thread limits (OMP_NUM_THREADS). By default a child process inherits the parent's environment. To override specific variables, copy the environment and modify it:

import os, subprocess

env = os.environ.copy()
env["LM_LICENSE_FILE"] = "27000@license-server"
env["OMP_NUM_THREADS"] = "4"
env["TMPDIR"] = "/scratch/jobs/crash_01"

result = subprocess.run(cmd, env=env, capture_output=True, text=True)

Two details are easy to get wrong. First, env= replaces the entire environment rather than merging — always start from os.environ.copy(). Second, on Windows, environment variable names are case-insensitive but Python dictionaries are not; if you set Path while the inherited environment has PATH, you can end up with two entries and unpredictable resolution. Use the exact casing the platform expects.

Working Directories

Solver output — .odb, .dat, .msg, .sta, restart files — is written relative to the process working directory, not the script's location. The cwd parameter sets this explicitly:

job_dir = "/scratch/jobs/crash_01"
result = subprocess.run(cmd, cwd=job_dir, env=env,
                        capture_output=True, text=True)

Without cwd, the child inherits the parent's directory. A monitoring script launched from /home/user/scripts would then scatter solver output into the scripts folder. Setting cwd per job keeps each run isolated and makes cleanup a matter of deleting one directory.

Note that cwd does not change how the executable is found. If abaqus is not on PATH, use an absolute path in cmd[0] or add its directory to the PATH entry of your copied environment.

Solver Launch: Inputs, Process, Outputs Parent Python Process cmd = [list of args] env = os.environ.copy() cwd = job_dir subprocess.run(...) CompletedProcess returncode / stdout spawn Child Solver Process abaqus job=crash_01 inherits env + overrides runs in cwd exit code 0 / non-zero stdout + stderr write Job Directory crash_01.odb crash_01.dat crash_01.msg crash_01.sta all paths relative to cwd

When You Need Popen

run blocks until the solver exits. For a crash simulation that may take hours, you often want to monitor it. Popen returns immediately with a handle:

proc = subprocess.Popen(cmd, cwd=job_dir, env=env,
                        stdout=subprocess.PIPE,
                        stderr=subprocess.STDOUT,
                        text=True)

try:
    stdout, _ = proc.communicate(timeout=3600)
except subprocess.TimeoutExpired:
    proc.kill()
    stdout, _ = proc.communicate()
    print("Job exceeded wall-clock limit and was killed")

print("exit code:", proc.returncode)

Key points: stdout=subprocess.PIPE captures output, and stderr=subprocess.STDOUT merges error text into the same stream so the log stays in chronological order. communicate() reads both pipes safely and avoids the deadlock that occurs when a child fills its pipe buffer while the parent is not reading. The timeout argument is what makes Popen worth the extra complexity — it gives you a hard wall-clock limit, which is essential when a solver hangs on a non-converging increment.

Interpreting the Result

Do not treat a zero exit code as proof of success. CAE solvers sometimes exit 0 after writing a partial result, and sometimes exit non-zero for benign warnings. A robust check combines three signals:

  1. Exit code — result.returncode == 0 is necessary but not sufficient.
  2. Log content — scan stdout for solver-specific markers such as THE ANALYSIS HAS COMPLETED SUCCESSFULLY or ***ERROR.
  3. Output artifacts — confirm the expected .odb or result file exists and is non-empty in cwd.
Practical pattern: wrap the launch in a small helper that returns a structured result — exit code, captured log, and a boolean success derived from all three signals. This keeps the monitoring layer in later chapters simple and consistent.

Common Failure Symptoms

  • FileNotFoundError — the executable is not on PATH; use an absolute path or fix the PATH entry in your copied environment.
  • License error in the log — LM_LICENSE_FILE was not set or was overwritten by an incomplete env dictionary.
  • Output files appear in the wrong folder — cwd was omitted, so the child inherited the parent's directory.
  • Process hangs indefinitely — no timeout was set on communicate(), or the solver is waiting on a license that will never be granted.
  • Truncated or interleaved log — stderr was not merged into stdout, so error messages are lost or out of order.

With these fundamentals in place — a list-based command, an explicitly copied and modified environment, a per-job working directory, and a result check that goes beyond the exit code — you have a reliable primitive on which the rest of the batch execution and monitoring system can be built.

← Previous 📖 Contents Next →