Why Garbage PDF Extraction Breaks RAG Retrieval Before It Starts

A document preprocessing API for LLM and RAG pipelines must extract readable text while preserving reading order, table relationships, headings, and paragraph boundaries. Garbage PDF extraction breaks retrieval before it starts because no embedding model or reranker can recover relationships that never reached the index. Reliable ingestion requires extraction tests against ground truth and accuracy gates before indexing.

Why PDFs are structurally incompatible with what RAG pipelines need

Consider a compliance contractor whose well-built RAG system passes internal QA, then gives wrong answers with confidence in production. An extractor has silently split an exception clause across two chunks, separating a rule from the condition that changes its meaning. The system retrieves the rule without its exception. Nothing crashes, and the answer sounds right. The failure began before retrieval had anything to search.

PDFs are a display format with unreliable semantic structure. They store glyph positions on a canvas, leaving extraction tools to reconstruct reading order and hierarchy from spatial clues. Position does not reliably establish logical sequence. A two-column research paper, a scanned invoice, and a slide deck exported to PDF therefore present different parsing problems. Yet the offline RAG sequence of parsing, chunking, embedding, and ingestion assumes that logically ordered text will emerge from each.

A parser that reads across the full page can interleave two columns, creating recognizable words in the wrong sequence. Flattening a table strips values of their row-column relationships, while missed headings turn sections into undifferentiated body text. Scans add lighting, rotation, skew, and handwritten annotations that degrade optical character recognition, or OCR. Characters drawn as curves can disappear entirely. These failures can all return usable-looking text, so the pipeline proceeds without an exception while the evidence loses its meaning.

How corrupted extraction propagates forward and becomes unrecoverable inside the index

Teams can spend weeks swapping embedding models, adjusting similarity thresholds, and tuning rerankers for marginal gains. The frustration has a mechanical explanation. An embedding model represents the text it receives, including any damage already done to that text. A better representation of a malformed passage is still a representation of the wrong evidence. Retrieval tuning cannot restore a relationship that extraction discarded.

The ICSE-SEIP '26 study on PDF parsing and chunking states that “if parsing produces incorrect structure… the retrieval component may fail to retrieve the information needed for correct answers”. Chunking can deepen that initial damage by splitting a table or separating a figure from its caption. A paragraph broken by a layout misread can become two orphaned halves, neither interpretable in isolation. The correct information remains visible in the PDF but is absent as a coherent unit in the index.

The damage can occur much later when parser or chunking logic changes and documents are reprocessed while old embeddings remain in the vector store. That version mismatch can look like model drift as retrieval recall deteriorates, sending developers toward another embedding-model change. RAGTrace distinguishes retrieval failure from generation failure after successful retrieval because the same wrong answer can demand different engineering work. An exact-match-zero result may point to extraction or indexing, or instead to prompting or model limitations. A single end-to-end score hides which layer went wrong and keeps teams working on the wrong one.

Why text accuracy scores mask structural destruction — and why engineers keep missing this

Across a benchmark of 800+ documents, the correlation between text accuracy and structural preservation was just 0.174. Parsers had high text accuracy but kept the structure intact at only roughly half that rate. Those are not interchangeable measures of quality. An output can score well on BLEU and edit similarity while leaving tables uninterpretable because the right characters survived in the wrong relationships.

The Peking University / Shanghai AI Lab survey names layout error cascading as one of the main ways these systems fail. A small layout detection error can disrupt later OCR and element parsing, corrupting an entire section. A controlled study of 302 questions makes the consequence concrete. Deep learning–based parsing beat rule-based parsing on nearly half the questions, tied on a large fraction, and lost on only a small minority, without changes to retrieval or generation. The improvement came from changing what the downstream system had to work with.

A single number says little unless it names the fields, document classes, and unit of measurement, because high character accuracy can still leave important fields wrong. Fraunhofer IAIS 2025 testing showed a substantial performance gap between clean invoices and scanned receipts on the same model. That comparison varied both document type and scanning status. In healthcare billing, accuracy can also fall sharply when the payer changes.

How to isolate extraction quality from retrieval quality before running any end-to-end eval

The practical first step is to build a golden extraction set before changing retrieval. Sample pages from the actual corpus, including tables, multi-column layouts, scans, footnotes, and nested lists, then manually annotate the correct text and structure. This establishes an independent reference for the parser. Run extraction on those pages and give character accuracy its own score, apart from the scores for heading detection, row-column integrity, reading order, and paragraph boundaries. Only after identifying the extraction layer’s contribution does an end-to-end evaluation become useful for diagnosing the remaining failures.

RAGTrace supports this separation through analysis that connects overall performance with retrieval relevance, generation fidelity, and cross-component interactions. It also enables knowledge-source tracing and identification of potential failure cases. That is the diagnostic distinction extraction auditing needs. If the parser never captured the supporting evidence correctly, a failed retrieval test is not a clean test of retrieval quality. If the evidence was extracted and retrieved correctly, the investigation can move to generation without treating every wrong answer as an ingestion defect.

Production evaluation also needs field-level ground truth. Structured fields such as year and state can be right almost every time, while free-form or context-dependent fields do much worse. The gap is the useful signal. For page-level confidence, an area-weighted approach multiplies each text block’s confidence by its bounding box area, preventing a small blurry element from distorting the score of an otherwise legible page. Confidence must also be calibrated and traceable to the page’s actual pixels. A score derived from a language model’s softmax distribution over generated tokens does not provide that grounding.

What should a PDF preprocessing API preserve, and which parser architecture can do it?

For documents with tabular data, table handling is the most important parser evaluation criterion. The architectural divide is between tools that understand layout and tools that primarily read characters. Complex documents need the former. Layout models find headings, tables, and reading zones before they extract text, keeping the links that give each word and value its meaning. The READoc benchmark (2025) found that tools with a deep learning pipeline had much higher overall scores than rule-based baselines.

PyMuPDF in default mode works well on documents with high-quality embedded text layers, but it needs OCR to extract text from true scans. PyMuPDF’s official documentation says that plain text extraction uses the order stored by the creator, which may differ from natural reading order. That limitation becomes costly on complex pages. Neural parsers address layout more directly, but the use of AI alone is not a guarantee of faithful output. Vision-language model parsers can hallucinate content in high-resolution, text-dense documents, so even a parser that extracts a wide range of content needs a separate check of its output.

LlamaParse illustrates why table fidelity deserves its own test. In a 48-cell sustainability report table, it captured values but placed them under the wrong dimensions, making the table uninterpretable. The numbers survived, but their meaning did not. Such column shifts can invalidate entire regional breakdowns even when every value appears somewhere in the output. Converting that output to structured JSON would not, by itself, repair the lost relationships.

Firecrawl illustrates a different architectural advantage through page classification before extraction. Its pdf-inspector analyzes font encodings, text operators, and image coverage on each page before choosing native extraction or neural OCR. Pages that do not need OCR avoid that expense. With this approach, it took much less time on average to process each page than with its previous pipeline. Because OCR dominates end-to-end execution time in these production pipelines, routing pages correctly can improve throughput more than optimizing the LLM parsing step.

Why extraction failures in agentic RAG are harder to recover from than in static pipelines

Agentic RAG interleaves retrieval and reasoning across multiple steps, allowing an early evidence defect to distort the rest of the reasoning path. Doctor-RAG documents how failures become more frequent as reasoning chains lengthen and interaction rounds increase. Errors accumulate rather than staying neatly isolated. Its analysis of agentic RAG that interleaves reasoning and action on HotpotQA identifies retriever, search, reasoning, and format errors, as well as dataset noise. With so many kinds of errors, failures that start in extraction can be hard to spot, and better single-step performance alone cannot make the whole workflow stable.

The framework separates diagnosis from repair rather than treating every failure as a reason to start again. DR-RAG diagnoses the trajectory, checks evidence sufficiency, classifies the failure, and locates its earliest point. It then applies tool-conditioned local repair while reusing conditionally valid reasoning prefixes and retrieved evidence. Blind reruns discard work that may still be sound. The extraction equivalent is targeted re-extraction of identified bad documents with a corrected parser, rather than re-embedding an entire corpus because one run produced damaged output.

Even a small error rate can produce dozens of daily failures requiring human review at large document volumes. High evaluation accuracy does not remove that workload. Validation layers, confidence thresholds, and human correction add more lines of defense when errors get past the parser. Learning from those corrections then gives the document system a way to improve, rather than reproducing a fixed error rate indefinitely.

Treating extraction accuracy as a hard engineering constraint, not a preprocessing choice

Production extraction needs measurable requirements, validation gates, and accountability, just like any other dependency governed by a service-level agreement. A vendor that cannot specify accuracy by field type and document class against ground truth leaves the pipeline operator absorbing the risk. Marketing claims do not allocate that risk. A contractual per-field accuracy SLA changes the arrangement by making the vendor responsible for the cost of misextraction. For a buyer choosing a document preprocessing API, that commitment is more useful than an undifferentiated accuracy percentage.

The build-versus-buy calculation must also include maintenance after launch. Rule-based and early neural parsers need more work as formats change and new edge cases turn up in production. Initial success is not the full cost. A system that silently accumulates extraction defects over months defers that cost until retrieval quality deteriorates. Specialist APIs that learn continuously from corrections can reduce the manual retraining and maintenance burden, but their value still depends on demonstrated performance on the target documents.

The engineering requirements follow from those failure modes. Before deployment, the golden set should include the hard cases in the corpus, and text accuracy and structural preservation should each have their own score. A per-field accuracy threshold should decide whether an extraction result goes into the index, while low-confidence fields go to human review. The vendor should demonstrate those requirements against the buyer’s documents and put them in an SLA.

Sensitive documents also need zero data retention. Financial documents, medical records, and legal contracts contain information that cannot be treated as incidental parser input. Verification should cover exact data definitions, endpoint matrices, runtime proof, access logs, and deletion evidence. Extraction belongs inside the production system’s accuracy and compliance boundaries, because the retrieval layer cannot compensate for defects that ingestion has already accepted.