2. Document ingestion pipelines

An NLP system is only as good as the text it receives. Before tokenization, embeddings, or retrieval can happen, a document ingestion pipeline must convert heterogeneous source files into a normalized, structured representation that downstream components can trust. This chapter examines how to parse the four most common enterprise formats — PDF, HTML, XML, and DOCX — while preserving three things that naive extraction destroys: layout, tables, and metadata.

Why Format-Aware Ingestion Matters

A flat string dump loses the signals that make technical documents useful. A maintenance procedure's numbered steps collapse into a paragraph. A specification table's row-column relationships vanish. A PDF's section hierarchy disappears. Downstream tasks such as question answering, clause retrieval, or compliance checking depend on these structures, so the ingestion stage must emit them explicitly rather than hoping a later model reconstructs them.

The practical target is a document object: an ordered list of typed blocks (heading, paragraph, list item, table, figure caption, code block) plus a metadata dictionary (title, author, creation date, source path, page or section anchors). Every parser below feeds into that same schema.

PDF: Layout Recovery from a Print Format

PDF stores glyphs at absolute coordinates with no semantic tags. Two extraction strategies exist. Text-layer extraction (PyMuPDF, pdfplumber, pdfminer.six) reads the embedded text objects and their bounding boxes; it is fast and exact but fails on scanned pages. OCR (Tesseract, PaddleOCR) rasterizes pages and recognizes pixels; it is necessary when no text layer exists but introduces character errors.

Layout preservation requires reconstructing reading order from coordinates. A workable heuristic: cluster text spans by their vertical position into lines, sort lines top-to-bottom, then within each line sort left-to-right. Detect columns by finding vertical whitespace gaps wider than a threshold (commonly 1.5× the median character width). For headings, compare font size against the document's body-text mode — spans exceeding it by a set ratio become heading candidates.

Tables are the hardest case. pdfplumber exposes page.find_tables(), which infers ruling lines and text alignment to produce a cell grid. When borders are absent, fall back to whitespace-gap detection. Always validate the extracted grid: a table with ragged row lengths usually signals a mis-detected boundary.

import pdfplumber

with pdfplumber.open("spec.pdf") as pdf:
    for page in pdf.pages:
        for table in page.find_tables():
            grid = table.extract()          # list of rows, each a list of cells
            assert len({len(r) for r in grid}) == 1, "ragged table"
        text = page.extract_text(layout=True)  # preserves column spacing

Metadata comes from pdf.metadata (title, author, producer) and from the document outline (pdf.outline) when bookmarks exist. Treat producer strings as unreliable — they often reflect the authoring tool, not the content.

HTML: Structure Is Already There

HTML is the easiest format because semantics are explicit. Parse with a tolerant parser (BeautifulSoup with lxml, or lxml.html directly) rather than regex. Walk the DOM and map tags to block types: <h1>–<h6> to headings, <p> to paragraphs, <ul>/<ol>/<li> to lists, <table> to tables, <pre><code> to code blocks.

Two traps deserve attention. First, boilerplate: navigation bars, footers, and cookie banners pollute the text. Strip elements matching known noise selectors (nav, footer, aside, class names containing "cookie" or "ad") before extraction. Second, table semantics: a <td> is meaningless without its column header. Read <th> cells and the scope attribute to attach headers to data cells, and honor colspan/rowspan when flattening.

from bs4 import BeautifulSoup

soup = BeautifulSoup(html, "lxml")
for tag in soup.select("nav, footer, aside, .cookie-banner"):
    tag.decompose()
for table in soup.find_all("table"):
    headers = [th.get_text(strip=True) for th in table.find_all("th")]
    for row in table.find_all("tr"):
        cells = [td.get_text(strip=True) for td in row.find_all("td")]
        if cells:
            record = dict(zip(headers, cells))

Metadata lives in <meta> tags, the <title> element, and JSON-LD blocks. Extract og:title, author, and datePublished where present.

XML: Schema-Driven Extraction

XML is structured by contract. Use lxml.etree or ElementTree and navigate by XPath rather than by position. The critical decision is namespace handling: DITA, DocBook, and JATS all use namespaces, and ignoring them causes silent empty results. Register prefixes explicitly.

from lxml import etree

tree = etree.parse("manual.xml")
ns = {"d": "http://docbook.org/ns/docbook"}
for para in tree.xpath("//d:para", namespaces=ns):
    print(para.text)

Preserve hierarchy by recording the ancestor path of each extracted node — this becomes your section anchor. Attributes such as id, xml:lang, and revision are first-class metadata and should be carried through.

DOCX: Unzipping the Package

A .docx file is a ZIP archive. The main content sits in word/document.xml; styles in word/styles.xml; metadata in docProps/core.xml. python-docx wraps this cleanly. Iterate document.paragraphs for text and document.tables for grids, but note that paragraphs and tables are interleaved in the body — to preserve order, walk document.element.body children directly.

from docx import Document

doc = Document("report.docx")
for child in doc.element.body.iterchildren():
    if child.tag.endswith("}p"):
        print("PARA:", child.text)
    elif child.tag.endswith("}tbl"):
        print("TABLE detected")

Heading levels come from paragraph styles (Heading 1, Heading 2), not from font size. Merged cells appear as gridSpan and vMerge elements and must be expanded before the grid is usable.

Choosing and Combining Parsers

Route by MIME type and file signature, not by extension alone. A pipeline typically dispatches: application/pdf to pdfplumber, text/html to BeautifulSoup, application/xml to lxml, and the DOCX MIME type to python-docx. Emit every result into the shared document-object schema so downstream code is format-agnostic.

Validation and Failure Symptoms

Every parser should be followed by a validation gate. Common symptoms and their causes:

With a normalized document object in hand, the next stage — chunking and indexing — can operate uniformly regardless of source format. The ingestion pipeline's job is to make that uniformity possible without discarding the structure that gives technical documents their meaning.