3. Industrial document ingestion pipelines

Transformer-based document intelligence models do not consume raw PDFs. Between a scanned invoice or a born-digital specification sheet and the token sequence that enters an encoder, there is an ingestion pipeline whose job is to convert heterogeneous, noisy, human-oriented artifacts into a normalized, machine-readable representation. In industrial settings this pipeline is where most production failures originate: a model that scores 0.95 F1 on a clean benchmark can drop below 0.70 when the upstream pipeline silently mis-rotates a page, drops a table column, or emits OCR text with broken reading order. This chapter treats ingestion as a first-class engineering subsystem rather than a preprocessing afterthought.

Pipeline stages and their contracts

A robust ingestion pipeline is a sequence of stages, each with an explicit input/output contract so that failures can be localized. The canonical stages are: acquisition (fetch the file and record provenance), format detection and routing (born-digital PDF vs. scanned image vs. mixed), rasterization and image preprocessing, OCR, layout parsing, metadata extraction, and normalization into a downstream schema. Each stage should emit both a payload and a quality signal (confidence, warnings, or a rejection flag) so that the orchestrator can decide whether to retry, escalate to a human, or pass the document through.

The single most consequential routing decision is born-digital vs. scanned. A born-digital PDF already contains a text layer with exact character positions; running OCR on it is wasteful and often degrades quality. The standard test is to extract the embedded text and measure the ratio of non-whitespace characters to page area. A practical heuristic: if a page yields fewer than roughly 100 characters per 100 cm² of content area, treat it as scanned or image-only and route it to OCR. Mixed documents (a scanned signature page bound into a digital contract) must be routed per page, not per file.

Rasterization and image preprocessing

OCR and layout models operate on pixels, so PDFs must be rasterized. Render at 300 DPI for text-heavy documents; 200 DPI is often sufficient for large-print forms but will lose small superscripts and thin table rules. Higher DPI increases latency roughly quadratically in memory, so 300 DPI is the usual industrial default, with 400 DPI reserved for documents containing sub-6pt text or dense tables.

Preprocessing then addresses the defects that scanning introduces. The most common operations, in order:

• Deskew: estimate the dominant text-line angle (e.g., via Hough transform or projection profile) and rotate to within ±0.5°. Residual skew above ~2° measurably degrades both OCR and layout detection.
• Denoising and binarization: adaptive thresholding (Sauvola or Otsu on local tiles) handles uneven illumination from flatbed and ADF scanners. Preserve grayscale for layout models that benefit from it; binarize only for the OCR engine if it expects binary input.
• Border and hole-punch removal: crop scanner black borders and mask punch holes, which otherwise produce spurious connected components.
• Despeckle: remove isolated salt-and-pepper noise smaller than a few pixels without eroding thin strokes.

OCR: engines, confidence, and reading order

Modern OCR is typically a detection-plus-recognition stack: a text detector (e.g., DBNet-style) produces word or line boxes, and a recognizer (CRNN or a Transformer-based sequence model) transcribes each crop. Two outputs matter downstream: the text and the per-token confidence. Do not discard confidence — it is the primary signal for triage. A common policy is to flag any page whose mean word confidence falls below a threshold (e.g., 0.80) or that contains more than a small fraction of words below 0.50.

Reading order is a separate problem from recognition. Detectors return boxes in arbitrary order; naive top-to-bottom, left-to-right sorting breaks on multi-column layouts, headers, and sidebars. Use a layout-aware ordering: cluster boxes into columns and regions first, then sort within each region. For Transformer models that consume a linearized token sequence, incorrect reading order is often more damaging than individual character errors, because it destroys the positional relationships the model relies on.

Layout parsing

Layout parsing assigns semantic roles — title, paragraph, table, figure, caption, header, footer, key-value pair — to regions. This is where document intelligence diverges from plain OCR. A layout model (e.g., a detection Transformer over page images) outputs region boxes and classes; a subsequent structure-recovery step reconstructs tables into cells and rows and links captions to figures.

Table recovery deserves special attention in industrial documents. A ruled table can be parsed from detected lines; a borderless table must be inferred from whitespace alignment and text baselines. Always validate the reconstructed table: check that row and column counts are consistent and that no cell is empty when the source clearly had content. A silently dropped column is a classic failure that propagates into downstream extraction as a missing field.

Metadata extraction and normalization

Metadata extraction captures document-level attributes: document type, source system, creation date, page count, language, and identifiers such as invoice number or PO number when they appear in a header. Some of this comes from the file itself (PDF /Info dictionary, XMP metadata), some from the content. Treat file-embedded metadata as untrusted — it is frequently stale or wrong — and prefer content-derived values when they conflict.

Normalization is the final contract with the model. It includes Unicode normalization (NFKC), whitespace collapsing, removal of control characters, date and number canonicalization, and language tagging. For a multilingual corpus, detect language per region rather than per document, since industrial packets often mix languages. The output of this stage is a structured record: an ordered list of blocks, each with text, role, bounding box, confidence, and page index, plus document-level metadata.

{
  "doc_id": "inv-2024-00871",
  "source": "scanner-adf-03",
  "pages": 2,
  "language": "en",
  "blocks": [
    {"page": 0, "role": "title", "text": "INVOICE", "bbox": [72,60,240,92], "conf": 0.98},
    {"page": 0, "role": "table", "rows": 12, "cols": 5, "conf": 0.91},
    {"page": 0, "role": "key_value", "key": "Invoice No.", "value": "A-10293", "conf": 0.95}
  ],
  "warnings": ["page 1 mean OCR confidence 0.74"]
}

Failure modes and triage

Three failure classes dominate production. Image defects (skew, glare, low contrast) show up as uniformly low OCR confidence and are fixed by re-scanning or stronger preprocessing. Structural defects (merged table cells, broken reading order) show up as plausible text in the wrong place and require layout-model or post-processing fixes. Semantic defects (correct text, wrong field assignment) are the hardest and usually need a validation rule or a human review queue. Instrument the pipeline to emit per-stage metrics — skew angle, mean confidence, table cell count, normalization rejections — so that a regression can be traced to a specific stage rather than blamed on "the model."

Putting it together: a reproducible ingestion run

Consider a batch of 500 scanned supplier invoices arriving as single-page TIFFs from an ADF scanner. A reproducible run proceeds as follows. First, acquisition records each file's hash and source device. Format routing classifies all pages as scanned (no text layer). Rasterization is skipped since the input is already an image; preprocessing deskews each page and applies Sauvola binarization. OCR produces word boxes and confidences. Layout parsing labels header, line-item table, and totals regions. Metadata extraction pulls the invoice number from the header region and the total from the totals region. Normalization canonicalizes dates to ISO 8601 and amounts to a fixed decimal format.

To verify the run, inspect three things: the distribution of mean OCR confidence across pages (a bimodal distribution usually indicates a subset of poorly scanned pages), the table cell counts (any invoice whose line-item table has fewer than two columns is suspect), and the fraction of records with a missing invoice number. If, say, 40 of 500 records lack an invoice number, check whether those pages share a source device or a header layout variant — that points to a routing or layout-model gap rather than an OCR problem. Typical symptoms and their likely causes: uniformly low confidence → image quality; correct text in wrong fields → layout or reading-order error; missing table columns → table structure recovery failure; garbled dates → normalization or locale handling. Each maps to a specific stage, which is exactly why the pipeline must be instrumented stage by stage rather than treated as a black box.

The output of a well-built ingestion pipeline is not merely text — it is a structured, confidence-annotated, normalized record that preserves the spatial and semantic relationships the downstream Transformer needs. Treating ingestion as an engineering subsystem with contracts, metrics, and triage rules is what separates a demo from a production document intelligence system.