4. Document ingestion and preprocessing for CAE

Retrieval-augmented assistants are only as good as the corpus they retrieve from. In CAE engineering, that corpus is unusually hostile to naive ingestion: solver manuals run to thousands of pages with deeply nested numbering, tutorials mix prose with GUI screenshots and command snippets, and legacy reports exist as scanned PDFs, exported HTML, or binary office files with inconsistent structure. This chapter covers how to turn that material into a clean, chunked, metadata-rich index that a retrieval system can actually use.

Why CAE documents break generic pipelines

A generic PDF-to-text tool assumes linear reading order. CAE documentation violates that assumption in predictable ways:

• Multi-column and side-by-side layouts. Solver theory manuals often place equations in one column and explanatory text in another. Naive extraction interleaves them into unreadable fragments.
• Equation-heavy content. Formulas rendered as vector graphics or embedded fonts extract as garbage or vanish entirely, leaving sentences like "the residual is computed as" with nothing after it.
• Deep heading hierarchies. A typical keyword reference uses levels such as 4.2.1.3, and the section number itself carries meaning. Losing it destroys retrievability.
• Tables as layout, not structure. Material properties, element type mappings, and convergence criteria are frequently tables with merged cells that flatten into meaningless token streams.
• Legacy formats. Older reports may be scanned images requiring OCR, or .doc/.xls files that need format-specific parsers.

The practical consequence: ingestion is not a single "convert to text" step. It is a pipeline with format detection, structure recovery, and quality gates.

Pipeline architecture

The diagram below shows the ingestion flow used throughout this course. Each stage produces an artifact that can be inspected and re-run independently, which matters because failures are usually localized to one stage.

Format detection and routing

Before any parsing, inspect each file to decide its route. The key question for PDFs is whether a text layer exists. A quick check with a PDF library returns the character count on a sample of pages; if a 20-page sample yields fewer than roughly 100 characters per page, the document is almost certainly scanned and must go through OCR. Native PDFs go straight to a layout-aware parser.

# Example: route PDFs by text-layer density
for f in corpus/*.pdf; do
  chars=$(pdftotext -f 1 -l 20 "$f" - | wc -c)
  pages=20
  density=$((chars / pages))
  if [ "$density" -lt 100 ]; then
    echo "OCR  $f  (density=$density)"
  else
    echo "NATIVE $f (density=$density)"
  fi
done

For HTML exports (common for online solver documentation), use a DOM parser rather than regex. For legacy .doc and .xls, convert to .docx/.xlsx first, then parse with a structured reader. Never feed binary office formats directly to a text extractor.

Structure recovery

Structure recovery is where retrieval quality is won or lost. Three elements matter most:

Heading hierarchy. Extract headings with their numbering and level. A section titled "4.2.1.3 Convergence criteria for nonlinear static" should be stored with section_number="4.2.1.3" and level=4. This lets the retriever filter by section depth and lets the assistant cite a precise location.

Tables. Detect table regions and convert them to structured rows. For a material property table, preserve column headers as field names. A flattened table like "Steel 210000 0.3 7850" is nearly useless; a structured row {material: Steel, E: 210000, nu: 0.3, rho: 7850} is directly retrievable.

Equations and figures. Equations rendered as images cannot be embedded as text. Two practical options: (a) run a formula-recognition model to produce LaTeX, or (b) store a caption plus a reference to the image region, and let the assistant retrieve the surrounding prose. Option (b) is cheaper and often sufficient, but the caption must be descriptive. If the source caption is "Figure 3," replace it with a generated caption derived from the surrounding paragraph.

Normalization

Normalization makes the corpus internally consistent so that a query about "Young's modulus" matches documents that write "E", "elastic modulus", or "modulus of elasticity." Apply these transformations:

• Unit normalization. Convert all quantities to a canonical unit set (SI by default) while preserving the original string in metadata. "210 GPa" and "210000 MPa" both become 210000000000 Pa with original="210 GPa".
• Symbol expansion. Expand ambiguous abbreviations on first occurrence per chunk, e.g., "E" → "Young's modulus (E)" when the context is material properties.
• Encoding repair. Fix mojibake from legacy exports (common with Greek letters and degree symbols).
• Whitespace and hyphenation. Rejoin words split across line breaks, collapse multiple spaces, and remove running headers/footers that repeat on every page.

Chunking strategy

Chunking determines what the retriever can return. For CAE documentation, fixed-size chunking is a poor default because it splits mid-equation and mid-table. Use semantic chunking instead:

1. Split at heading boundaries first. A section is the natural unit.
2. If a section exceeds the token budget (commonly 512–1024 tokens), split at paragraph boundaries, never inside a table or equation block.
3. Add overlap of one to two sentences between adjacent chunks so that context spanning a boundary is not lost.
4. Prepend a breadcrumb to each chunk: [Solver: ANSYS Mechanical] [Section: 4.2.1.3] [Topic: Nonlinear static convergence]. This dramatically improves retrieval for queries that mention the solver or topic but not the exact wording.

A practical token budget for CAE text is 600 tokens with 80-token overlap. Tables should be chunked as whole units even if they exceed the budget, because a half-table is worse than an oversized chunk.

Metadata schema

Every chunk should carry metadata that supports filtering and citation. A minimal but sufficient schema:

Quality gates

After ingestion, run automated checks before the index goes live. Three gates catch most problems:

• Coverage. Compare the number of extracted sections against the table of contents. A mismatch above 5% indicates parsing failures.
• OCR confidence. Chunks with ocr_confidence below 0.85 should be flagged for manual review or excluded from the index.
• Chunk sanity. Reject chunks that are empty, contain only whitespace, exceed 2000 tokens, or consist mostly of non-alphanumeric characters (a sign of equation or table extraction failure).

Handling legacy reports

Legacy CAE reports deserve separate treatment because they are often the most valuable and the least structured. A scanned report from a 2010 project may contain hand-annotated results, non-standard terminology, and no digital text layer. The workflow:

1. OCR with a layout-preserving engine, not a plain text OCR. Preserve column and table boundaries.
2. Manually correct the first few pages to calibrate the OCR engine's dictionary with project-specific terms (material grades, element names, solver keywords).
3. Tag the report with project metadata: project_id, analyst, date, solver_version. This enables retrieval scoped to a specific project history.
4. Store the original page image alongside the extracted text so that any answer can be traced back to the source scan.

Ingestion is not a one-time task. As new solver versions are released and new reports are written, the pipeline must be re-runnable incrementally. Design it so that a single document can be re-ingested without rebuilding the entire index, and so that every chunk retains a traceable path back to its source page.