4. Document layout understanding with Transformers

Traditional text-only models treat a document as a flat sequence of tokens. That assumption collapses the moment you face a real invoice, a purchase order, or a P&ID drawing, where meaning depends heavily on where a token sits on the page. A total amount printed in the bottom-right corner means something different from the same string appearing in a header. Layout-aware Transformers solve this by fusing three modalities: the text tokens, the visual appearance of the page, and the 2D spatial coordinates of each token. This chapter examines the LayoutLM family, DocFormer, and the general principles of multimodal fusion that make them work on forms, invoices, and technical drawings.

Why 2D Position Embeddings Matter

In a standard BERT encoder, each token receives a 1D positional embedding indexed by its order in the sequence. LayoutLM (2019) extended this by adding a 2D position embedding derived from the token's bounding box. For a token with box coordinates (x0, y0, x1, y1), the model computes separate embeddings for the x-axis and y-axis, each split into a left/right or top/bottom component, then sums them with the token embedding and the 1D sequence embedding. The result is that two identical words at different page locations produce different hidden states.

The coordinate convention matters and must be fixed before training. LayoutLM uses the top-left corner as the origin, with x increasing to the right and y increasing downward, and normalizes all coordinates to a 0–1000 integer scale relative to the page width and height. This normalization makes the model resolution-independent, so a 300 DPI scan and a 72 DPI render of the same page yield comparable embeddings. If you switch to a bottom-left origin (common in some CAD exports), you must convert before feeding the model, or the spatial signal becomes inverted and accuracy drops sharply.

From LayoutLM to LayoutLMv2 and v3

LayoutLMv1 relied on an external OCR engine and only used text plus coordinates. LayoutLMv2 (2020) added a visual backbone (a ResNet or similar CNN) that produces an image feature map, which is then fused into the Transformer through a cross-attention mechanism. This means the model sees the actual pixels — font weight, table lines, logos, stamps — not just the recognized characters. LayoutLMv2 also introduced a spatial-aware self-attention bias, so attention scores are modulated by the relative distance between token boxes.

LayoutLMv3 (2022) unified text and image under a single patch-based scheme: the page image is split into patches (like ViT), and text tokens are aligned to those patches. This removed the dependency on a separate CNN backbone and made pretraining objectives consistent across modalities — masked language modeling, masked image modeling, and word-patch alignment. For industrial documents, v3's advantage is robustness to noisy OCR: even when a character is misrecognized, the visual patch still carries the correct glyph shape.

DocFormer and the Attention over Modalities

DocFormer (2021) took a different route. Instead of a single fused stream, it uses multi-modal self-attention where text, visual, and spatial features each have their own projection, and the attention mechanism learns to weight them per token. Its key innovation is a spatial-aware attention that encodes relative position between every pair of tokens, plus a shared CNN backbone that produces both the visual features and the spatial embeddings. For technical drawings — where a dimension label may be far from the geometry it annotates — this long-range spatial attention is particularly valuable.

The practical trade-off across these models is straightforward. LayoutLMv1 is cheapest to run and works well when OCR is clean and layout is simple. LayoutLMv2/v3 and DocFormer cost more compute but tolerate noisy scans, complex tables, and graphical elements. For a typical invoice pipeline, LayoutLMv3-base fine-tuned on 2,000–5,000 labeled pages usually reaches 92–96% token-level F1 on key fields, while v1 on the same data typically plateaus around 85–89%.

Applying This to Forms, Invoices, and Technical Drawings

The workflow is consistent regardless of model choice. First, run OCR (Tesseract, PaddleOCR, or a cloud API) to obtain tokens with bounding boxes. Second, normalize coordinates to the 0–1000 grid. Third, tokenize with the model's tokenizer, aligning subword pieces back to their source boxes. Fourth, fine-tune with a token-classification head using BIO labels such as B-KEY, I-KEY, B-VALUE, I-VALUE, B-TABLE_CELL.

A concrete labeling example for an invoice line:

Token          Box (x0,y0,x1,y1)   Label
Invoice        (120, 340, 200, 372) B-KEY
No.            (205, 340, 240, 372) I-KEY
:              (245, 340, 252, 372) O
INV-2024-0871  (260, 340, 420, 372) B-VALUE
Total          (120, 620, 180, 652) B-KEY
:              (185, 620, 192, 652) O
4,250.00       (200, 620, 320, 652) B-VALUE

After training, inference produces per-token labels. A post-processing step groups consecutive B-KEY/I-KEY spans and pairs them with the nearest B-VALUE span on the same row (using y-coordinate overlap). This yields structured JSON such as {"invoice_no": "INV-2024-0871", "total": "4,250.00"}.

Practical Checklist for a Layout Model Pipeline

The most common failure mode is a mismatch between the coordinate system used during labeling and the one used at inference. If your annotation tool exports boxes in PDF points but your OCR returns pixels, the 2D embeddings will be inconsistent and the model will underfit. Always log a few sample boxes and visually overlay them on the page image before committing to a full training run. Once the coordinate contract is stable, LayoutLMv3 or DocFormer will reliably outperform text-only baselines on any document type where spatial position carries semantic weight.