4. File and directory operations for CAE job management

A crash simulation campaign rarely fails because of a single bad solver run. It fails because the wrong include file was picked up, a restarted job overwrote a converged result, or a 40 GB archive was copied without preserving timestamps. In CAE automation, file and directory operations are not housekeeping — they are part of the solver's input contract. This section covers the four commands that carry most of that load (find, xargs, rsync, tar) and the directory layout that makes them safe to use.

A layout that encodes model versions and run identity

Before any command is useful, the tree must be predictable. A workable convention separates immutable model data from mutable run output, and makes the run directory name self-describing:

crash_project/
├── models/
│   └── body_intrusion/
│       ├── v01_baseline/
│       │   ├── main.key
│       │   ├── includes/
│       │   │   ├── mat_steel_bpw.k
│       │   │   ├── mat_foam_bumper.k
│       │   │   └── mesh_rail_front.k
│       │   └── manifest.sha256
│       └── v02_thickness_update/
│           ├── main.key
│           ├── includes/
│           └── manifest.sha256
├── runs/
│   └── body_intrusion/
│       └── v02_thickness_update/
│           ├── 2024-06-11_0930_50kmh_full/
│           │   ├── input/        # symlinks to models/v02...
│           │   ├── work/         # solver scratch, d3dump, d3plot
│           │   ├── logs/
│           │   └── status
│           └── 2024-06-11_1415_50kmh_restart/
└── archive/

Three rules make this layout automation-friendly. First, model directories are read-only after a manifest is written; a run never edits its own inputs. Second, run directories are created, never reused — the timestamp plus load case in the name guarantees uniqueness, so a restart cannot silently overwrite a completed run. Third, inputs are referenced by symlink from runs/.../input/ back to the model version, so the solver sees a flat working directory while provenance stays traceable through readlink.

find: selecting inputs and cleaning scratch safely

find is the selection engine. Two patterns dominate CAE work: gathering all include decks for a manifest, and pruning scratch files older than a threshold.

# Hash every include deck of a model version, sorted for reproducibility
find models/body_intrusion/v02_thickness_update -type f -name '*.k' -print0 \
  | sort -z | xargs -0 sha256sum > models/body_intrusion/v02_thickness_update/manifest.sha256

# Delete solver scratch older than 7 days, but never touch d3plot or logs
find runs/body_intrusion -type f \
     \( -name 'd3dump*' -o -name '*.tmp' -o -name 'fort.*' \) \
     -mtime +7 -delete

Note the deliberate use of -print0 and xargs -0. CAE paths frequently contain spaces (part names like rail front) and occasionally non-ASCII characters. Newline-delimited pipelines break on those; NUL-delimited ones do not. Also note that -delete is applied only after an explicit name filter — a bare find ... -mtime +7 -delete on a run tree will happily remove a converged d3plot that took six hours to produce.

For a dry run, replace -delete with -print and inspect the list first. This costs seconds and prevents the most expensive class of automation mistake.

xargs: parallel execution across load cases

Once find produces a list of run directories, xargs turns it into parallel solver invocations. The key discipline is to pass one item per command and to keep the argument count bounded:

find runs/body_intrusion/v02_thickness_update -maxdepth 1 -mindepth 1 -type d -print0 \
  | xargs -0 -n1 -P4 -I{} bash -c '
      run="$1"
      echo "running $run" >> "$run/logs/queue.log"
      cd "$run/work" && ls-dyna i=../input/main.key memory=800m \
        > ../logs/solver.out 2>&1
      echo $? > "$run/status"
    ' _ {}

Here -n1 guarantees one directory per invocation, -P4 caps concurrency at four (match this to your license tokens, not to your CPU count), and -I{} substitutes the path into the shell snippet. The exit code is written to status, which later becomes the filter for post-processing. Without that file, a failed run is indistinguishable from a successful one until someone opens the log.

rsync: moving results without corrupting them

Results are large and often transferred to a scratch or archive volume. rsync is preferred over cp because it preserves timestamps, resumes partial transfers, and can verify content:

rsync -av --partial --checksum \
      --include='*/' --include='d3plot*' --include='binout*' \
      --exclude='*' \
      runs/body_intrusion/v02_thickness_update/2024-06-11_0930_50kmh_full/ \
      /mnt/archive/body_intrusion/v02_thickness_update/2024-06-11_0930_50kmh_full/

The trailing slash on the source is significant: it copies the contents of the run directory into the destination, not the directory itself. The include/exclude ordering matters too — --include='*/' must come first so rsync descends into subdirectories before the catch-all --exclude='*' prunes everything else. --checksum forces content comparison rather than the faster size-and-mtime heuristic; use it for final archival, skip it for routine staging.

tar: packaging a run for handoff

When a run must be delivered to a colleague or attached to a report, tar produces a single reproducible artifact. Exclude scratch and keep the manifest inside the archive:

tar --exclude='work/d3dump*' --exclude='work/*.tmp' \
    -czf archive/body_intrusion_v02_50kmh_full_20240611.tar.gz \
    -C runs/body_intrusion/v02_thickness_update \
    2024-06-11_0930_50kmh_full

Using -C to change directory before listing the member keeps paths relative, so the archive extracts cleanly anywhere. Verify before shipping:

tar -tzf archive/body_intrusion_v02_50kmh_full_20240611.tar.gz | head
tar -tzf archive/body_intrusion_v02_50kmh_full_20240611.tar.gz | wc -l

If the member count is zero or the listing shows absolute paths, stop and rebuild the archive.

Putting it together: a run lifecycle

The diagram below shows how the four commands chain across a single run's life, from model selection through archival, with the status file as the decision point.

Verification and common failure symptoms

After a batch completes, three checks confirm the pipeline behaved:

• Manifest integrity: sha256sum -c models/.../manifest.sha256 must report OK for every include. A mismatch means a model file changed after the run started — the results are not attributable to the recorded version.
• Status coverage: find runs/... -name status | wc -l should equal the number of run directories. A missing status means the xargs invocation was killed before writing it.
• Archive completeness: tar -tzf member count should match the file count under the run directory minus excluded scratch.

The through-line is that every command here is a filter or a carrier, and each one needs an explicit boundary: find needs a name or time predicate, xargs needs a concurrency cap and a per-item invocation, rsync needs a trailing-slash decision and an include order, tar needs a -C anchor and a verification pass. Set those boundaries once in a wrapper script, and the directory layout does the rest of the work — a run's provenance is readable from its path, its outcome from its status file, and its inputs from its manifest.