2. Data collection and preprocessing for failure analysis

Reliability studies live or die by the quality of the underlying data. A failure analysis that draws on incomplete test logs, misaligned sensor timestamps, or ambiguous maintenance codes will produce confidence intervals and hazard-rate estimates that look precise but are fundamentally untrustworthy. This section covers the four primary data sources used in engineering test failure investigations — test logs, sensor streams, maintenance records, and the cleaning pipeline that unifies them — and shows how to transform raw, heterogeneous records into an analysis-ready reliability dataset.

Test Logs: Structured Failure Event Records

Test logs are the backbone of any failure-time dataset. Each row typically represents one test unit and captures its identity, start time, stop time, and terminal condition. The critical distinction is between a failure event (the unit failed at a known time) and a right-censored observation (the unit survived until the test was suspended or the observation window closed). Confusing these two destroys the validity of any subsequent Weibull or exponential fit.

A well-formed test log contains at minimum: unit ID, test start timestamp, event timestamp, event type (failure / suspension / removal), failure mode code, and environmental conditions. Consider a vibration endurance test on 20 actuator assemblies. The log might look like this:

unit_id,start_ts,event_ts,event_type,mode_code,ambient_C,rpm
A-01,2024-03-01T08:00,2024-03-01T14:22,failure,BRG-SPALL,42,3600
A-02,2024-03-01T08:00,2024-03-01T19:05,failure,SEAL-LEAK,44,3600
A-03,2024-03-01T08:00,2024-03-02T08:00,suspension,,41,3600
...
A-20,2024-03-01T08:00,2024-03-01T11:47,failure,BRG-SPALL,43,3600

Here, A-03 is right-censored at 24 hours — it had not failed when the test ended. The mode code BRG-SPALL indicates bearing spalling, while SEAL-LEAK points to a different failure mechanism. These codes matter because reliability models are typically fitted per failure mode; mixing modes into one pool violates the assumption of a single underlying failure distribution.

Sensor Streams: High-Frequency Condition Data

Sensor streams provide the physical evidence of degradation. Accelerometers, thermocouples, pressure transducers, and current probes generate time-series data at rates from 1 Hz to 50 kHz. Unlike test logs, sensor streams are not event-based — they are continuous and must be reduced to features that correlate with failure progression.

Common preprocessing steps for sensor streams include:

• Resampling and alignment: Different sensors sample at different rates. Resample all channels to a common time base (e.g., 1-second intervals) using interpolation, and align timestamps to a single clock reference. Clock drift between data loggers is a frequent source of apparent anomalies.
• Feature extraction: Raw waveforms are rarely used directly. Compute time-domain features (RMS, kurtosis, peak-to-peak), frequency-domain features (FFT peak amplitude, spectral centroid), or envelope analysis for bearing diagnostics.
• Outlier detection: Sensor dropouts, spikes, and saturation produce values that are physically impossible. A temperature reading of 999 °C on a test rig running at 45 °C is a sensor fault, not a failure event.

For the actuator test, an accelerometer mounted on the bearing housing might show a gradual increase in RMS vibration from 0.8 g to 3.2 g over 14 hours before the spall event. That trend is the degradation signal; the test log records only the endpoint. Combining both gives a complete picture: the log tells you when it failed, the sensor stream tells you how it degraded.

Maintenance Records: Field Failure and Repair Data

In field reliability studies, maintenance records replace controlled test logs. These records are messier: they are written by technicians, use inconsistent terminology, and often lack precise failure times. A work order might say "replaced bearing, excessive noise" without specifying the operating hours at failure.

Key preprocessing tasks for maintenance records include:

• Text normalization: Map free-text descriptions to standardized failure mode codes. "Bearing noisy," "bearing rough," and "bearing vibration high" should all map to BRG-DEGRADED.
• Time reconstruction: If only the repair date is known, estimate the failure time using the previous inspection date or the midpoint between last-known-good and repair. Document the estimation method, because it introduces uncertainty.
• Duplicate and merge detection: Multiple work orders may describe the same physical failure event. Merge them using unit ID, timestamp proximity, and failure mode consistency.

The Cleaning Pipeline: From Raw Records to Analysis-Ready Data

Data cleaning for reliability studies follows a repeatable sequence. The goal is a single tidy table where each row is one unit, with columns for time-to-event, event indicator (1 = failure, 0 = censored), failure mode, and covariates.

A practical pipeline for the actuator example:

1. Ingest and parse: Load test logs, sensor feature tables, and maintenance records into a common dataframe. Parse timestamps to ISO 8601 and convert to a consistent timezone.
2. Validate ranges: Flag any sensor value outside physical limits (e.g., rpm < 0, temperature > 200 °C). These become missing values, not failures.
3. Compute time-to-event: For each unit, calculate duration = event_ts - start_ts in hours. For censored units, use the suspension time.
4. Encode event indicator: Set event = 1 for failures, event = 0 for suspensions and removals.
5. Join covariates: Attach sensor features (e.g., peak RMS before failure) and environmental conditions (ambient temperature, rpm) to each unit row.
6. Check consistency: Verify that no unit has a negative duration, that failure modes are from the approved code list, and that censored units have no failure mode assigned.

After cleaning, the dataset might contain 20 rows with 14 failures and 6 censored observations. The failure modes split into 9 bearing spalls and 5 seal leaks. This is now ready for a Weibull fit or a competing-risks model.

Practical Considerations and Common Pitfalls

Three issues recur in real reliability studies. First, time synchronization: if the test log clock and the sensor logger clock differ by even a few minutes, the computed time-to-event will be wrong. Always record a synchronization event or use NTP on all acquisition systems. Second, censoring transparency: every removed or suspended unit must be flagged. A unit that was removed for an unrelated reason (e.g., test rig reconfiguration) is not a failure and must not be coded as one. Third, mode coding discipline: establish the failure mode taxonomy before data collection begins. Retroactively assigning modes from free-text notes introduces subjective bias.

The cleaned dataset is the foundation for everything that follows — parameter estimation, confidence bounds, and reliability predictions. Investing effort in collection and preprocessing pays off directly in the credibility of the final analysis. In the next stage, this dataset will be used to fit lifetime distributions and assess whether the observed failures are consistent with the assumed physics of failure.