4. Data cleaning and preprocessing for manufacturing data

Manufacturing data rarely arrives analysis-ready. A typical yield dataset is assembled from several systems that were never designed to agree with one another: process equipment logs events on its own clock, the MES records lot genealogy at step boundaries, metrology tools report measurements keyed by wafer or unit ID, and final test returns pass/fail bins hours or days later. Before any yield model can be trusted, these streams must be reconciled into a single, coherent table where every row represents a well-defined unit of production and every column has a known meaning. This chapter covers the four preprocessing problems that dominate semiconductor and discrete-manufacturing yield work: missing values, outliers, time alignment, and lot tracking.

Establishing the Unit of Analysis and the Join Key

Cleaning decisions only make sense once the grain of the table is fixed. In wafer fabs the natural grain is often the wafer (identified by lot ID plus wafer slot), while in assembly and test it may be the unit or the panel. Choose the grain first, then verify that every source can be mapped to it. The join key is usually a composite such as (lot_id, wafer_id) or (lot_id, unit_id). A frequent failure is silent key mismatch: the MES pads lot IDs to a fixed width ("L1234") while the equipment log trims them ("1234"). Normalizing keys — trimming whitespace, upper-casing, and applying a consistent zero-pad rule — prevents rows from being dropped during joins without any error being raised.

Handling Missing Values

Missingness in manufacturing data is rarely random. A sensor may be offline during a specific maintenance window, a metrology step may be skipped for certain product routes, or a value may be recorded as -999 or NaN depending on the tool vendor. The first task is to distinguish three cases:

• Structurally missing — the measurement does not exist for that route (for example, an overlay measurement only performed on certain layers). These should be excluded from imputation and handled by route-aware modeling.
• Missing at random — a sporadic dropout unrelated to the process outcome. These are candidates for imputation.
• Missing not at random — absence correlates with the outcome (a tool that only logs when it alarms). Imputing these can inject bias and should be flagged rather than filled.

For genuinely imputable numeric process parameters, median imputation within a lot is often more defensible than a global mean, because process conditions drift lot to lot. A practical rule is to impute only when a parameter is missing for less than a threshold fraction of units within a lot (commonly 10–20%) and to add a companion boolean column such as param_missing so the model can learn from the missingness pattern itself. Never impute the target yield value.

Detecting and Treating Outliers

Outliers in manufacturing fall into two categories that require opposite responses. Erroneous outliers come from sensor glitches, unit-conversion mistakes, or data-entry errors and should be corrected or removed. Genuine extreme values may be the very excursions that caused yield loss and must be preserved. Distinguishing them requires process context, not just statistics.

A robust screening approach combines a statistical rule with a physical plausibility check. For each parameter, compute the median and the median absolute deviation (MAD), which is far less sensitive to extreme values than the standard deviation:

MAD = median(|x_i - median(x)|)
robust_z = 0.6745 * (x_i - median(x)) / MAD

Flag values with |robust_z| > 3.5 as candidates. Then apply a physical bound: a chamber temperature of 850 °C on a tool whose recipe maximum is 400 °C is an error, whereas a 5 °C deviation during a spike is a real event worth keeping. Only the intersection of "statistically extreme" and "physically implausible" should be corrected. For genuine excursions, retain the value and add an excursion_flag column so downstream analysis can separate their effect.

Time Alignment

Process steps occur at different times and different sampling rates. Aligning them means assigning each measurement to the correct production unit and ordering events correctly. Two conventions must be fixed and applied consistently:

• Timestamp source and timezone. Equipment clocks may drift or run in local time while the MES uses UTC. Convert everything to a single reference (UTC is standard) before any comparison.
• Alignment direction. For each unit, associate each process step with the measurement whose timestamp falls within that step's window. Define windows explicitly, e.g. a step starting at t0 with duration d owns measurements in [t0, t0 + d).

A common pitfall is clock skew between tools: a downstream step may appear to occur before its upstream step. Detect this by checking that step timestamps are monotonically increasing along the route for each unit; violations indicate skew or mis-keyed data and must be resolved before feature engineering. When sampling rates differ, resample to a common cadence using forward-fill for slow-changing states and interpolation only for smooth physical signals — never interpolate across a step boundary.

Lot Tracking and Genealogy

Lot tracking links every unit to the sequence of tools, chambers, and recipes it passed through. This genealogy is what allows yield loss to be attributed to a specific chamber or a specific split. Two operations matter most:

1. Reconstructing the route. Sort each unit's events by aligned timestamp and verify the sequence matches the expected process flow. Missing steps or out-of-order steps signal data problems.
2. Propagating attributes. Carry tool ID, chamber ID, recipe ID, and operator shift forward so that each unit row contains the full context of its history.

When lots are split or merged (common in assembly), maintain a parent-child mapping so that a yield problem in a child lot can be traced back to the shared parent operation. Without this, root-cause analysis stops at the merge point.

A Practical Validation Checklist

After preprocessing, run a small set of assertions before the data enters any model. These catch the most common silent failures:

Treat these checks as code, not as a one-time manual review. A short validation script that runs on every new data pull will surface a broken join or a shifted clock long before it corrupts a yield model. The goal of preprocessing is not to make the data pretty — it is to make every downstream conclusion traceable to a real unit, a real tool, and a real point in time.