1. Test-CAE Correlation Workflow Setup and Data Ingestion

Why the ingestion stage decides the CORA outcome

Every CORA scorecard is only as trustworthy as the data that enters it. Before a single metric is computed, the test and CAE signals must share a common time base, a common unit system, and a verified channel mapping. If any of these three is wrong, the correlation result will be wrong in a way that no amount of metric tuning can repair — a 5 ms lag or a millimetre/metre mismatch will silently depress the score and send the team chasing a modelling problem that does not exist. This case establishes a reproducible ingestion workflow that produces a single, auditable correlation dataset from two independent sources: physical test time histories and CAE simulation output.

The two source objects and their native formats

In a typical crash or vibration correlation study the two inputs look like this:

• Test side: a multi-channel time-history file (for example an ISO-MME / RPC III style directory, or a single .mat/.csv export) sampled at a fixed rate such as 10 kHz, with channels named by measurement location (ACC_SILL_L, ACC_TUNNEL_R, FORCE_BARRIER). Units are declared per channel in a header or sidecar file.
• CAE side: a solver output (for example LS-DYNA nodout/binout, or a post-processed .csv) sampled at the solver time step, often 1 µs–1 ms, with node/element IDs rather than physical names. Units follow the solver's consistent unit system (e.g. mm–ms–kg–kN).

The ingestion task is to reconcile these two into one table where row i corresponds to the same physical instant for both signals, and column headers carry the same physical meaning and unit.

Step 1 — Load and inventory both sources

Load each source without resampling yet, and record what you actually received. The inventory is the audit trail that later lets you prove the scorecard used the right data.

import numpy as np
import pandas as pd

# --- Test side: one CSV per channel, first column = time in seconds ---
test = {}
for ch in ["ACC_SILL_L", "ACC_TUNNEL_R", "FORCE_BARRIER"]:
    df = pd.read_csv(f"test/{ch}.csv")
    test[ch] = {"t": df["time_s"].to_numpy(),
                "y": df[ch].to_numpy(),
                "unit": df.attrs.get("unit", "unknown")}

# --- CAE side: single CSV, time in milliseconds, node IDs as columns ---
cae_raw = pd.read_csv("cae/nodout_export.csv")
cae = {}
for node, ch in [("1001", "ACC_SILL_L"), ("1002", "ACC_TUNNEL_R"),
                 ("2001", "FORCE_BARRIER")]:
    cae[ch] = {"t": cae_raw["time_ms"].to_numpy() / 1000.0,   # -> seconds
               "y": cae_raw[node].to_numpy(),
               "unit": "mm/s2" if ch.startswith("ACC") else "kN"}

inventory = pd.DataFrame([
    {"channel": ch,
     "test_dt_s": np.median(np.diff(test[ch]["t"])),
     "cae_dt_s":  np.median(np.diff(cae[ch]["t"])),
     "test_unit": test[ch]["unit"],
     "cae_unit":  cae[ch]["unit"]}
    for ch in test
])
print(inventory)

The printed inventory is the first checkpoint. A test dt of 1e-4 s and a CAE dt of 1e-6 s tells you immediately that the CAE signal is 100× oversampled relative to test — expected, and handled later by resampling (case 2), not here. What matters at this stage is that both time vectors are monotonic, start near the same event trigger, and that the unit columns are populated rather than unknown.

Step 2 — Normalise units to one consistent system

Pick one target system and convert everything into it. A practical choice for correlation work is SI-adjacent: seconds, metres per second squared, newtons. Convert CAE accelerations from mm/s2 to m/s2 by dividing by 1000, and forces from kN to N by multiplying by 1000. Do the conversion explicitly and log the factor applied per channel — never rely on an implicit assumption buried in a plotting call.

UNIT_TO_SI = {
    ("mm/s2", "m/s2"): 1e-3,
    ("kN",    "N"):    1e3,
    ("m/s2",  "m/s2"): 1.0,
    ("N",     "N"):    1.0,
}

def to_si(channel, unit, y):
    target = "m/s2" if channel.startswith("ACC") else "N"
    factor = UNIT_TO_SI[(unit, target)]
    return y * factor, target

for ch in cae:
    cae[ch]["y"], cae[ch]["unit"] = to_si(ch, cae[ch]["unit"], cae[ch]["y"])
for ch in test:
    test[ch]["y"], test[ch]["unit"] = to_si(ch, test[ch]["unit"], test[ch]["y"])

Step 3 — Build the common time base and verify alignment

Define a single correlation time vector that spans the overlapping window of both signals, then interpolate each source onto it. Interpolation here is only for placing both signals on identical sample instants; it is not the filtering/resampling step that case 2 covers. Use linear interpolation for the ingestion check and keep the original arrays intact for later processing.

t_start = max(test["ACC_SILL_L"]["t"][0], cae["ACC_SILL_L"]["t"][0])
t_end   = min(test["ACC_SILL_L"]["t"][-1], cae["ACC_SILL_L"]["t"][-1])
dt_common = 1e-4                       # 10 kHz common grid
t_common = np.arange(t_start, t_end, dt_common)

aligned = {"t": t_common}
for ch in test:
    aligned[f"test_{ch}"] = np.interp(t_common, test[ch]["t"], test[ch]["y"])
    aligned[f"cae_{ch}"]  = np.interp(t_common, cae[ch]["t"],  cae[ch]["y"])

corr_df = pd.DataFrame(aligned)
print(corr_df.shape, corr_df.isna().sum().sum())   # expect (N, 7) and 0

Two checks confirm the common time base is sound. First, corr_df.isna().sum().sum() must be zero — any NaN means one source did not cover the window. Second, compare the peak time of each channel pair: for a well-triggered dataset the test and CAE peaks should fall within a few milliseconds of each other. A large offset is a trigger or lag issue (case 3), not an ingestion bug, but it must be recorded now so it is not mistaken for a modelling error later.

Step 4 — Channel mapping and the audit record

The mapping between physical test channels and CAE node IDs is the most error-prone part of ingestion, because a swapped left/right channel produces a plausible-looking but wrong scorecard. Freeze the mapping in an explicit table and store it alongside the dataset. The table below is a complete, fillable example for a three-channel study:

The Peak Δt column is the difference between the test peak time and the CAE peak time on the common grid. Values of a few milliseconds are normal for a correctly triggered dataset; a value of tens of milliseconds points to a trigger offset that case 3 will address. Record it now so the ingestion record is complete and the later lag-correction step has a documented starting point.

Failure symptoms and what they mean

• NaN in corr_df: the overlap window was computed from one channel only; recompute t_start/t_end as the intersection across all channels.
• Peak Δt of hundreds of ms: the CAE time column was already in seconds and was divided by 1000 again, or the test trigger fired late. Check the raw time vector endpoints before touching the mapping.
• Acceleration magnitudes off by exactly 1000×: a unit conversion was applied twice or skipped. The factor column in the audit table makes this visible immediately.
• Left/right channels correlate suspiciously well: the node mapping is swapped. Cross-check against the CAE model's node coordinate list, not against the signal shape.

Completion criteria

The ingestion stage is complete when: (1) corr_df contains no NaN and has one column pair per mapped channel; (2) every channel pair shares the target unit recorded in the audit table; (3) the peak Δt for each pair is recorded and within the expected trigger tolerance; and (4) the audit table and dataset are stored together as the single input to the next stage. Only then is the data ready for CORA evaluation.