Output Contracts Between Extraction APIs and LLM Context Windows

For document preprocessing APIs in LLM and retrieval-based generation pipelines, valid JSON is not enough. The right API must preserve field meaning, report calibrated uncertainty, cite source locations, distinguish missing information from unreadable information, and enforce review rules before data enters an LLM context window. An output contract makes those requirements explicit, but pipeline quality depends on its weakest joint.

Why the output contract is the right place to start — and why starting there is not enough

Practitioners building extraction pipelines have converged on a sound starting point. Set out the output contract before writing extraction code. That gives the extraction system and its downstream consumer a shared agreement about what an acceptable response looks like.

But a design is not an enforcement mechanism. A contract that looks airtight can fail between the raw document and the populated context window, with one layer accepting an error another layer should have caught. The mistake is treating a schema file as proof that the agreement holds in production.

What an output contract actually consists of, and why each component is load-bearing

An output contract is a layered agreement between an extraction API and the system consuming its results. It governs more than the shape of JSON. Schema conformance defines field names, nesting, and the number of records allowed in arrays. Field typing defines what those values mean, including whether a value is a date, a currency amount with its denomination, or an identifier whose characters must be preserved.

Confidence scores show how uncertain each extracted field is. For those scores to work as contract terms, their link to error rates must be measured on real documents. Source citations show the evidence behind the values. Each citation points to a verifiable source location, such as a page number and bounding box.

Null semantics define the different kinds of missing information. They distinguish a field that is absent, one that is present but illegible, and one the system did not attempt to extract. Validation gates then define which results proceed automatically, which require human review, and which are rejected. ExtractBench evaluates schema-guided extraction through value accuracy, record completeness, word- and page-level grounding, and cost together. These are separate dimensions of extraction quality.

How schema conformance breaks under document variance, and why strict typing alone cannot fix it

Accepting a schema parameter is not proof of reliable extraction. Schema support means the API accepts the structure it is asked to use. Schema reliability means it returns conformant output against complex schemas across the documents it actually receives. The gap grows as schemas become more complex, but document variance exposes a further problem. A response can satisfy the structure while assigning text to the wrong fields.

Layout changes are enough to break that assignment. A parser that handles a standard invoice can misread a multi-column or rotated version because its assumptions about where text belongs no longer hold. Forms that mix typed fields with handwritten annotations add another source of confusion. Insurance documents that combine tables with dense prose make the boundaries between records harder to resolve.

ExtractBench identifies scan noise, handwriting, hierarchical headers, cross-page continuation, and irregular tables as document challenges. Amazon's PARSE research also identifies ambiguous field descriptions and unclear entity boundaries as triggers for hallucination despite structured-output support. The engineering response is to give schemas unambiguous descriptions, explicit entity boundaries, and example values, then test them against the hardest variants in the production corpus. A clean sample invoice is not that test.

Why field typing is where silent errors enter the pipeline

An invoice number coerced to an integer loses its leading zeros. The response can still be perfectly valid JSON. Yet the resulting identifier matches no record in the system of record, because the extraction layer changed the value while making it look cleaner. This is a semantic error. The structure gives the downstream system no reason to stop.

Other conversions create the same false appearance of correctness. A currency amount reduced to a bare number loses the distinction between USD and EUR. A date left in a locale-specific string can support two readings, leaving the consumer to choose one silently. A boolean field that was never stated can become false rather than null, turning “not stated” into “explicitly denied.”

OCR noise can corrupt the value before normalization begins. “Invoice #INV-2024-001” becoming “lnvoice #lNV-2O24-OO1” substitutes lookalike characters inside an identifier. A regex or template can then fail to interpret the field correctly while a malformed string passes onward as a valid identifier. The contract therefore needs semantic commitments as well as primitive types. “String” describes storage, while “identifier with its original characters preserved” describes the obligation the downstream application depends on.

Why uncalibrated confidence scores are worse than no confidence scores

A made-up purchase order number can come with high confidence. When the source is missing or illegible, an extraction system may infer a value from nearby text, the filename, or the email body. The model can assign high token probability to that answer even though it did not extract it from the document. A high score then becomes permission to accept a fabrication. That is worse than leaving uncertainty explicit.

The score must mean something outside the model. Calibration tests its relationship to actual errors on a holdout of real documents. OCR confidence is about the recognition step, while model token probabilities show how likely the generated output was. Neither, by itself, establishes that the accepted field is correct.

This calls for per-field selective risk control. Fields are accepted only when the measured error rate among accepted results stays within a target threshold on a calibration split. A raw confidence cutoff does not establish that guarantee. The practical approach is to start conservatively with high automatic-acceptance thresholds and lower review thresholds, then use reviewer actions to refine them by field type and document class. Grounding is still needed because a confidence score can only be checked independently if it has source evidence.

What source citations do in the contract, and what happens when they are absent

A correct extraction and a hallucinated value can look identical in JSON. Without a source pointer, the output offers no way to tell them apart. Someone must return to the document and find the evidence manually. At production volume, rereading whole documents defeats the purpose of the automation. A citation is therefore a required part of the contract.

That distinction directly affects human review. A reviewer needs the source region that holds the value to fix it without searching the whole document. ExtractBench reports word-level and page-level grounding F1 separately from value accuracy because correct values can still come with unverifiable citations.

Grounding also has a document-security role. The CrackedPDFs benchmark describes hidden-text attacks as a mismatch between what a document shows and what downstream extraction reads. When the PDF text layer contains hidden prompts, tracing a value only to that layer does not establish that it came from visible page content. Visual grounding ties the evidence to the rendered page instead. The contract requirement is a verifiable source location for every extracted value, expressed as a page number and bounding box or equivalent.

Null semantics — the contract term that most schemas leave undefined

A purchase order number inferred from a filename can conceal a missing field. Instead of admitting that the document supplies no readable value, the extraction system returns a plausible one. Without a source citation, the substitution is hard to detect. This is the inverse of a missing extraction. The system fills a gap that the contract should preserve.

A single null cannot explain every gap. Production documents require at least three distinct states, which are absent, illegible, and not-attempted. An absent due date does not establish that no due date applies, although a downstream LLM may read it that way. An illegible date needs a different remediation path from a genuinely absent date. A field that was not searched for says nothing about what the document contains.

The contract should represent those distinctions explicitly. An enumerated absence type with the members absent, illegible, and not-attempted belongs beside the value, and every downstream point that uses it needs rules for how to handle it. ExtractBench credits a correct null when a source field is blank and scores an omitted key as an explicit null. That scoring rule treats blank answers as part of value accuracy. It does not replace the operational distinction between why a value is missing, which the consuming application still needs to act correctly.

Validation gates as the enforcement layer the other joints depend on

A gate based only on schema conformance misses semantic type errors and ambiguous null states. One based on raw confidence assumes the scores reflect accuracy. One without grounding cannot tell a source-backed answer from a made-up value. Gates make the contract enforceable only when they evaluate its separate obligations together.

The production architecture has three routes. Automatic acceptance requires passing type checks, calibrated confidence above the acceptance threshold, and a verifiable citation before the value enters the context window. Human review takes values that are structurally valid but whose confidence is below that threshold or whose citation is missing. When grounding is available, the source region can be highlighted for fast correction. A missing citation prevents that shortcut and leaves the reviewer with more work.

Rejection takes values that fail type checks, have ambiguous null states, or break business rules. It must return a machine-readable reason for the failure. A bare pass-or-fail flag does not give the application enough information to retry extraction, reformat a result, or escalate the case. Exception routing is part of enforcement.

Gate quality also depends on the test population. A handful of difficult documents cannot represent the production tail, where remaining failures concentrate after easy cases are handled. Reviewer corrections need to feed back into extraction and calibration so that thresholds improve over time. A static gate tuned at deployment degrades as document populations shift. Acceptance rules need continuing evidence.

Why the weakest joint determines the quality of the entire pipeline, not the strongest model

Upgrading the model is an understandable response to poor output. It can also target the wrong failure. A frontier model reading unreadable source material can produce high-confidence tokens about OCR noise. The broken joint is between the document and extraction. More capable generation does not make the source legible.

The same dependency runs through a retrieval-based generation pipeline. Bad parsing leads to poor chunking, which weakens embeddings and produces irrelevant retrieval that feeds hallucinations. Repairing retrieval does not repair the parsing error that started the chain. Aggregate accuracy can obscure this because fields differ in reliability. A strong average leaves unanswered whether the particular field controlling a downstream decision is dependable.

Useful diagnosis asks which joint has the lowest reliability and how it fails. Answering it requires per-field measurement against ground truth. When vendors report only aggregate accuracy without per-field service commitments, customers cannot tell where the risk they are taking lies. For buyers choosing a document preprocessing API, schema support is therefore only a starting requirement. A better basis for choosing is proof that each contract obligation holds on the documents the pipeline will use.