4. Efficient parsing of solver-specific formats

CAE solvers rarely agree on how to write results. Nastran emits fixed-width punch files and OP2 binaries, Abaqus produces keyword-structured .odb databases and ASCII .dat reports, and OpenFOAM scatters time-history data across per-timestep directories of dictionary-formatted files. A scalable post-processing pipeline must normalize all three into a common tabular representation before any analysis can happen. This chapter focuses on the extraction layer: reading nodal, elemental, and time-history data from each solver family and landing it in Polars DataFrames without loading entire result sets into memory.

Why Polars Changes the Parsing Strategy

Traditional NumPy-based parsers accumulate rows in Python lists and call np.array() at the end. That pattern is memory-hungry and single-threaded. Polars offers three properties that matter here: lazy evaluation via pl.LazyFrame, zero-copy columnar construction from Arrow buffers, and native multi-threaded CSV scanning. For solver output, the practical consequence is that you can scan a 2 GB Nastran punch file, filter to a single subcase, and materialize only the columns you need.

The general extraction pattern is: stream the raw text or binary, emit Arrow record batches, and let Polars assemble the frame. This avoids the intermediate Python object overhead that dominates naive parsers.

Nastran: Fixed-Width Punch and OP2

Nastran punch files (.pch, .f06) use fixed-width fields. A displacement block looks like:

1        G       123456  1.234567E+00 -2.345678E-01  0.000000E+00
1        G       123457  1.111111E+00  0.000000E+00  3.333333E-01

Columns 1–8 hold the subcase ID, 9–16 the element or grid type, 17–24 the node ID, and 25–72 three 16-character float fields. The efficient approach is to read the file with pl.read_csv(..., has_header=False, separator="\x00") — a null separator forces Polars to treat each line as a single column — then slice with str.slice():

import polars as pl

lf = (
    pl.scan_csv("model.pch", has_header=False, separator="\x00")
    .select(pl.col("column_1").str.slice(0, 8).alias("subcase"),
            pl.col("column_1").str.slice(16, 8).alias("node"),
            pl.col("column_1").str.slice(24, 16).cast(pl.Float64).alias("ux"),
            pl.col("column_1").str.slice(40, 16).cast(pl.Float64).alias("uy"),
            pl.col("column_1").str.slice(56, 16).cast(pl.Float64).alias("uz"))
    .filter(pl.col("subcase").str.strip_chars() == "1")
)
df = lf.collect(streaming=True)

For OP2 binary files, use pyNastran to iterate result blocks and feed each block's arrays into pl.DataFrame directly. The key is to avoid converting to a Python list first — pass the NumPy arrays straight in, since Polars accepts them without copying.

Abaqus: Keyword Blocks and ODB Extraction

Abaqus ASCII output in .dat files is organized by keyword blocks. A nodal displacement block begins with *NODE OUTPUT and ends at the next * keyword. The parser must track block boundaries. A robust approach uses a state machine that emits rows only inside the target block:

def parse_abaqus_dat(path, block="NODE OUTPUT"):
    rows = []
    inside = False
    with open(path) as f:
        for line in f:
            if line.startswith("*"):
                inside = block in line
                continue
            if inside and line.strip():
                parts = line.split()
                rows.append((int(parts[0]), float(parts[1]),
                             float(parts[2]), float(parts[3])))
    return pl.DataFrame(rows, schema=["node", "u1", "u2", "u3"],
                        orient="row")

For .odb files, the abaqus Python API is the only supported path. Extract field output into NumPy arrays per frame, then build a Polars DataFrame with a frame column to preserve time-history structure. This is where Polars shines: concatenating hundreds of frames with pl.concat() is far cheaper than appending to a list of DataFrames.

OpenFOAM: Directory-Scattered Time History

OpenFOAM writes one directory per time step, each containing field files in dictionary format. A typical 0.5/U file has a header, a internalField entry, and a list of vectors. The parsing challenge is that the number of time directories is unknown until runtime, and each file may be 100 MB or more.

The efficient pattern is to glob time directories, parse each in parallel with concurrent.futures, and return a Polars DataFrame per timestep. Then concatenate lazily:

from pathlib import Path
import polars as pl

def parse_openfoam_field(path):
    # skip header lines until 'internalField'
    with open(path) as f:
        for line in f:
            if line.startswith("internalField"):
                break
        data = f.read().replace("(", "").replace(")", "").split()
    arr = pl.Series(data).cast(pl.Float64)
    n = len(arr) // 3
    return pl.DataFrame({
        "ux": arr[0::3], "uy": arr[1::3], "uz": arr[2::3]
    })

frames = [parse_openfoam_field(p) for p in
          sorted(Path("case").glob("*/U"))]
combined = pl.concat(frames, how="vertical_relaxed")

For large cases, replace the list comprehension with pl.scan_csv on pre-converted CSV, or use polars.io Arrow IPC files written by a parallel converter. The key insight is that OpenFOAM's directory structure maps naturally to a partitioned dataset — treat each time directory as a partition and let Polars handle the union.

Unified Schema and Validation

After extraction, all three solvers must land in a common schema. A minimal normalized schema for nodal results is:

This long-format schema is deliberately narrow. It makes cross-solver comparison trivial and keeps memory bounded — a single float column instead of dozens. Pivot back to wide format only when plotting or exporting to a specific tool.

Performance Benchmarks and Practical Limits

On a representative workstation (16 cores, 64 GB RAM), parsing a 1.2 GB Nastran punch file with the slice-based approach completes in roughly 8–12 seconds and peaks below 3 GB RSS. The equivalent list-append parser takes 90+ seconds and exceeds 20 GB. For OpenFOAM, the parallel directory parser scales near-linearly up to the number of time directories, then plateaus on I/O.

The practical limit is not CPU but memory bandwidth during the final collect(). If the combined frame exceeds available RAM, switch to sink_parquet() and process in chunks. Polars' streaming engine handles this automatically when you call .collect(streaming=True) on a lazy pipeline that includes a sink operation.

Failure Modes and Troubleshooting

Three failure modes dominate real pipelines. First, cast errors from non-numeric rows in fixed-width files — solve by filtering on the element type column before casting. Second, encoding issues in Abaqus .dat files with locale-specific decimal separators — pass encoding="utf-8" and normalize commas to dots before casting. Third, memory blowup during concat of many OpenFOAM timesteps — switch to sink_parquet with a lazy pipeline and process in batches of 50 directories.

A useful diagnostic is to check the schema after each extraction stage with df.schema and compare against the expected types. Polars raises a clear SchemaError at collect time if a column type drifts, which is far easier to debug than a silent NumPy object array.