4. Data cleaning and validation

Raw CAE results are rarely ready for reporting the moment a solver finishes. A CFD run may leave NaN in cells where turbulence quantities diverged, a structural analysis may export displacements in millimeters while the report template expects meters, and a mesh may contain negative-volume elements that silently corrupt every derived quantity downstream. This chapter treats data cleaning and validation as a first-class engineering step: a repeatable pipeline that runs between "solver finished" and "plot generated." The goal is not cosmetic tidiness but defensible numbers — every value that reaches a report should be traceable, dimensionally correct, and computed on a mesh that passed basic quality gates.

Why Validation Must Precede Visualization

Postprocessing tools are permissive by design. pandas.read_csv() will happily ingest a column of empty strings, numpy.mean() will propagate NaN without complaint, and a contour plot will render a mesh with inverted elements as if nothing were wrong. The failure mode is therefore silent: a mean outlet temperature computed over 3% missing samples, or a pressure drop reported in the wrong unit system, looks perfectly plausible on a slide. The discipline is to insert explicit checks that fail loudly before any aggregation or plotting happens.

A practical validation stage answers three questions in order: (1) Is the data complete enough to aggregate? (2) Are all quantities in a consistent, declared unit system? (3) Is the mesh geometrically valid for the quantities being derived? Each question maps to a distinct set of checks, and each check should produce a machine-readable verdict rather than a human eyeball.

Handling Missing Values

Missing data in CAE exports appears in several disguises: empty CSV fields, sentinel values such as -999 or 1e30, literal NaN from a diverged solver, and rows dropped entirely by a failed export. The first task is detection, not imputation.

import pandas as pd
import numpy as np

SENTINELS = [-999.0, -9999.0, 1e30]

df = pd.read_csv("outlet_probe.csv")
df = df.replace(SENTINELS, np.nan)

report = df.isna().sum().to_frame("missing_count")
report["missing_pct"] = 100 * report["missing_count"] / len(df)
print(report.sort_values("missing_pct", ascending=False))

Once the missing pattern is visible, the decision to drop, interpolate, or flag depends on the physical meaning of the column. A time-history probe with a few isolated gaps can be safely interpolated; a spatially distributed field with a contiguous missing patch usually indicates a solver or export problem that imputation would only hide. A defensible rule set:

• Drop the sample when missingness is isolated (< 1% of rows) and the quantity is used only for a global statistic such as a mean.
• Interpolate only along a monotonic independent variable (time or a spatial coordinate), and record the interpolation in the report metadata.
• Reject the dataset when missingness exceeds a threshold you declare up front — 5% is a common default for scalar monitors, but the threshold should be justified by the sensitivity of the reported quantity.

Unit Consistency

Unit errors are the most expensive class of postprocessing mistake because they survive every numerical check. A displacement of 2.5 is valid whether the unit is millimeters or inches; only the label distinguishes them. The robust approach is to attach units to data at the point of ingestion and convert explicitly, rather than trusting column headers.

import pint
ureg = pint.UnitRegistry()

def to_si(value, unit_str):
    return (value * ureg(unit_str)).to_base_units().magnitude

# Example: mixed-unit export from two solvers
disp_mm = 2.5
disp_m  = to_si(disp_mm, "mm")      # -> 0.0025
stress_psi = 14500.0
stress_pa = to_si(stress_psi, "psi")  # -> 9.997e7

For a full dataset, declare a canonical unit system (SI is the safe default) and convert every column on load. Keep the original unit string in a sidecar dictionary so the report can state, for example, "displacements converted from mm to m." Two checks catch most unit problems before they propagate:

• Magnitude sanity: compare each converted column against a physically plausible range for the application. A steel stress above 2 GPa or a displacement above the model's bounding-box diagonal is almost always a unit error.
• Dimensional consistency: verify that derived quantities carry the expected dimensions. A pressure drop computed as rho * v**2 / 2 must reduce to pascals; if it reduces to pascals per meter, a length term is missing.

Mesh Quality Checks

Mesh quality is not a postprocessing concern in the abstract — it directly determines which derived quantities are trustworthy. The checks below are cheap to run on an exported mesh and catch the defects that most often invalidate results.

A minimal mesh audit reads the element volume array and the quality metrics the mesher already computed, then reports counts and locations of violations:

vol = mesh.cell_volumes          # shape (n_cells,)
ortho = mesh.orthogonality       # 0..1, higher is better

bad_vol = np.where(vol <= 0)[0]
bad_ortho = np.where(ortho < 0.1)[0]

print(f"negative/zero volume cells: {len(bad_vol)}")
print(f"low-orthogonality cells:    {len(bad_ortho)}")
if len(bad_vol):
    print("first offending cell ids:", bad_vol[:10])

When violations are found, the correct response depends on severity. A handful of low-orthogonality cells in a far-field region may be acceptable if a mesh-sensitivity study shows the reported quantity is insensitive to them. Any negative-volume cell is a hard stop: the mesh must be repaired or regenerated before results are used. Record the audit outcome — cell counts, worst metric values, and the decision — in the report's validation section so a reviewer can see the basis for accepting the results.

Putting It Together: A Validation Record

The three gates should emit a single structured record that travels with the report. A minimal schema, stored as JSON alongside the figures, makes the validation auditable without re-running the pipeline:

{
  "dataset": "outlet_probe.csv",
  "rows_ingested": 12000,
  "missing": {"T_outlet": {"count": 42, "pct": 0.35, "action": "interpolated"}},
  "units": {"displacement": "mm -> m", "pressure": "psi -> Pa"},
  "mesh": {"cells": 184320, "negative_volume": 0,
           "min_orthogonality": 0.14, "max_aspect_ratio": 87.3},
  "gate_decision": "PASS",
  "reviewer": "jdoe",
  "timestamp": "2025-01-15T09:42:00Z"
}

Two habits make this pipeline durable. First, run it as a script, not as ad-hoc notebook cells, so the same checks execute on every dataset. Second, fail loudly: raise an exception when a hard gate is violated rather than printing a warning that scrolls off screen. A validation stage that can be ignored will be ignored, and the cost of a unit error or an inverted element discovered after a report is issued is far higher than the cost of a pipeline that refuses to produce a figure until the data earns it.