2. Transformer fundamentals for document AI

Industrial document intelligence systems rarely operate on clean prose. They ingest scanned invoices, multi-page contracts, maintenance work orders, engineering drawings with title blocks, and semi-structured forms where meaning depends on spatial layout as much as on word order. The Transformer architecture is the backbone of most modern systems that handle this material, but applying it effectively requires understanding what each component actually does — and where the standard formulation needs adjustment for documents.

Self-Attention: The Core Mechanism

Self-attention lets every token in a sequence attend to every other token, computing a weighted representation based on learned relevance. For an input sequence of n tokens with hidden dimension d, the mechanism projects each token into three vectors: a query Q, a key K, and a value V. The attention output is:

Attention(Q, K, V) = softmax(QKᵀ / √d_k) · V

The scaling factor √d_k prevents the dot products from growing large enough to push the softmax into saturated regions where gradients vanish. In document AI, this mechanism is what allows a model to link a line item's quantity field to its unit price and to the column header several rows above — relationships that sequential models struggle to capture across long spans.

The practical cost is quadratic complexity: attention over n tokens requires O(n²) operations. A single dense page of text might tokenize to 800–1,200 tokens, which is manageable. A 40-page contract can exceed 30,000 tokens, at which point full self-attention becomes expensive. This is why production systems often combine sliding-window attention, sparse attention patterns, or hierarchical chunking with cross-chunk aggregation.

Multi-Head Attention

A single attention operation captures one notion of relevance. Multi-head attention runs h attention operations in parallel, each with its own learned projections, then concatenates and linearly combines the results. With h = 8 and d = 512, each head operates on 64-dimensional subspaces.

For documents, different heads empirically specialize: some track syntactic dependencies, others align table cells to headers, and others capture numeric patterns such as currency symbols preceding amounts. This division of labor is not explicitly programmed — it emerges from training — but it explains why multi-head attention consistently outperforms single-head attention on structured extraction tasks.

Positional Encoding

Self-attention is permutation-invariant: without positional information, "Invoice 2024-001" and "001 2024-001 Invoice" produce identical representations. Positional encoding injects order information. The original Transformer uses sinusoidal functions:

PE(pos, 2i)   = sin(pos / 10000^(2i/d))
PE(pos, 2i+1) = cos(pos / 10000^(2i/d))

Modern document models frequently replace or augment this with learned positional embeddings, relative position biases (as in T5), or rotary position embeddings (RoPE). For documents, 1D sequence position is often insufficient. Layout-aware models such as LayoutLM add 2D bounding-box coordinates to the token embedding, so a token's representation encodes not just where it appears in the reading order but where it sits on the page. This matters when a form has two columns: a naive 1D encoding interleaves them, while a 2D encoding preserves the spatial grouping that a human reader uses.

Encoder-Decoder Variants

The original Transformer is an encoder-decoder model. The encoder processes the input sequence bidirectionally; the decoder generates output autoregressively, attending to both its own prior outputs and the encoder's representations via cross-attention.

For pure extraction tasks — identifying vendor name, invoice number, line items — encoder-only models with a token classification head are usually the most efficient choice. For tasks that require generating structured output from a document image, encoder-decoder models like Donut skip OCR entirely and map image patches directly to a JSON sequence. Decoder-only models are increasingly used for document QA, where the prompt contains retrieved document chunks and the model generates an answer.

Tokenization for Industrial Documents

Tokenization is where document AI diverges most sharply from general NLP. Industrial documents contain part numbers (AX-4471-B), measurements (12.5 mm ±0.02), dates in inconsistent formats, and tables where whitespace carries meaning.

Byte-pair encoding (BPE) and WordPiece, the standard subword tokenizers, handle rare words by splitting them into frequent subword units. But a part number like AX-4471-B may fragment into AX, -, 44, 71, -, B — losing the atomic identity of the identifier. For extraction tasks, this fragmentation can hurt accuracy because the model must reassemble the identifier from pieces.

Practical strategies include:

• Domain vocabulary augmentation: Add frequent part numbers, unit symbols, and field labels to the tokenizer vocabulary before fine-tuning. This reduces fragmentation and shortens sequences.
• Character-level fallback for numeric fields: Some systems use a separate character-level encoder for numeric spans, preserving digit-level precision for amounts and quantities.
• Layout-aware token boundaries: In table extraction, tokens are often defined per cell rather than per whitespace-delimited word, so that a cell containing "12.5 mm" is treated as a single unit.

Putting It Together: A Document Processing Pipeline

The following SVG illustrates how these components connect in a typical industrial document intelligence pipeline, from raw input through tokenization, encoding, and task-specific output heads.

The diagram shows two paths: an encoder-only path for extraction tasks (top row) and an encoder-decoder path for generation tasks (bottom branch). In practice, the choice between them depends on whether the target output is a set of labeled spans or a generated sequence. For a work order that must be converted into a structured JSON record, an encoder-decoder model with cross-attention is often the better fit. For a form where each field is a classification decision, an encoder-only model with a token classification head is simpler and faster.

The configuration parameters at the bottom are starting points, not universal constants. A model processing single-page invoices can use a max sequence length of 512 and 6 encoder layers. A model handling multi-page contracts may need 12–24 layers and a sparse attention mechanism to keep memory within bounds. The right values depend on document length distribution, the number of distinct field types, and the available GPU memory during training.