The Developer's Negotiation Checklist for Financially Backed OCR Accuracy Commitments

A contractual per-field accuracy SLA for document extraction APIs needs defined measurement rules, a test on the buyer’s documents, and a financial remedy for failure. This developer’s negotiation checklist explains what must be written into the contract before an OCR accuracy claim becomes a commitment the buyer can enforce.

Why OCR accuracy claims are rarely what vendors mean them to be

Vendors and buyers often use “accuracy” to describe different results. A vendor may mean text recognition on clean pages, while the buyer means getting the right values from the documents that enter production. Both can discuss the same percentage without agreeing on what was measured. The gap is not just a wording problem. The buyer pays for errors that the headline figure never promised to catch.

High accuracy across individual fields can still leave many documents with at least one wrong value as the number of fields grows. A document-level figure also needs a definition of what makes a document correct. Without that rule, it conceals how errors accumulate across fields.

Recognizing clean printed text is different from extracting a structured amount, and both differ from interpreting a liability cap or a complex date condition. One headline number hides those differences in difficulty. An accuracy number has financial weight only when the contract spells out how it is measured.

How production failure modes differ from what the demo showed

In insurance, generative-model extractors sometimes return a value that does not appear in the document, such as an effective date inferred from context when no effective date is present. The value looks correct to downstream systems. This is fabrication, and a clean-document demo does not establish how often it happens.

In mortgage and financial services, an incorrect application field can trigger an automated rejection. That consequence makes traceability and accountability necessary for assessing accuracy. Healthcare exposes a different weakness in the proposed remedy. Human review can fail through alert fatigue and automation bias, so adding a reviewer does not by itself make an accuracy commitment dependable.

Routing also changes which accuracy figure applies. Mature hybrid pipelines use deterministic OCR for clean cases, layout-aware models for structured variation, and vision or language models for genuinely ambiguous layouts. Each tier handles a different extraction problem. A blended score hides that distinction. The negotiation must begin with field-specific measurement.

What a defensible accuracy definition must specify before negotiation starts

Start by requiring a written definition of per-field accuracy. The contract must identify the fields in scope and how each is scored. This is the first contractual requirement. It shows an error in a financial amount that a score for the whole document could hide.

Next, name the comparator to use for each field type. Stickler from AWS Labs uses normalized Levenshtein distance as its default string comparator, providing a character-level way to assess string similarity. Numeric fields need a tolerance-based comparator with the permitted percentage defined, while dates need semantic calendar equivalence. These choices belong in the agreement because they define a correct returned value.

Require a labeled holdout set drawn from the buyer’s actual document types as the ground truth corpus. Vendor-prepared samples and internal test sets controlled by the vendor do not meet that requirement. Exclude confidence scores as substitutes for ground-truth evaluation. Otherwise, reported confidence can be tuned upward without proving that the extracted values are any more accurate.

Finally, set thresholds by field category and risk. A financial amount may require a higher threshold than a notes or comments field because the cost of error differs. The legal language must preserve that asymmetry. Ask for the unit of measurement, named comparators, ground truth corpus, confidence-score exclusion, and field-specific thresholds to be part of the contractual definition.

The test the vendor must run on your documents before you sign

Select the validation sample from your real document corpus. It must cover the document types, layouts, scan qualities, and vendors or counterparties that the production pipeline will handle. The vendor must not select or prepare it.

Have the vendor run that sample using the agreed evaluation methodology. Ask for results for every field in scope, and have standard fields reported apart from non-standard and edge-case fields. Do not accept an aggregate score in their place. The split must expose whether strong performance on easy fields is masking weak performance on the fields that need the most scrutiny.

Require the report to state the exact methodology applied so that it can be compared with the negotiated definition. Use the resulting field-level figures as the baseline for drafting the SLA. A refusal to run the test, or an insistence on controlling the sample, is a red flag. It leaves the proposed commitment unproven on your document population, regardless of the vendor’s intentions or published results.

What terms make a contractual per-field accuracy SLA financially enforceable?

Write into the SLA the minimum accuracy floors for each field category and document type. Separate high-risk fields, including financial amounts and dates that trigger obligations, from lower-risk fields. Do not accept one aggregate threshold for all of them. The floor must describe the minimum acceptable performance for the extraction work covered by the contract.

Define a breach as a measurable, documented failure to meet the applicable floor within the agreed measurement period. Specify how the failure is determined and who holds the ground-truth labels. Those details spell out how the buyer proves a claim. If they are left open, an extraction error that can be measured can lead to a fight over whose evaluation counts.

Attach a remedy to that breach definition. A number without a remedy is a target. The exclusive-remedy clause should not limit the buyer to service credits when accuracy failures cause downstream financial harm. Ask for carve-outs for consequential damages when a specific extraction error can be traced to a specific loss, because credits are not enough for every failure.

Set the claim deadline around the time needed to detect errors downstream. Atlassian’s published SLA requires credit claims within fifteen days after the end of the calendar month in which the alleged failure occurred. That is a claim-window example. A short window can prevent recovery for errors discovered later, so negotiate enough time for those errors to surface.

Examine scope exclusions with the same care as remedies. Scan resolution, language, and handwriting restrictions must not remove your most common document types from coverage. Specify which exclusions are acceptable before signing. Also ask for payment tied to correctly extracted pages rather than raw page volume, because that model puts the vendor’s financial interest on the same side as the buyer’s accuracy requirement.

Data retention and security terms that belong in the same negotiation

Negotiate data handling with accuracy. A pipeline that keeps sensitive documents in logs, support tools, or training sets brings risks that an accuracy remedy does not cover. For regulated document processing, the relevant compliance stack includes SOC 2 Type II, ISO 27001, GDPR, and HIPAA for healthcare. The negotiation must cover what happens to document data during and after extraction.

Require documented zero data retention architecture where that is the agreed standard. Under that design, document content exists only in volatile memory for the extraction session rather than in a database, log file, or backup. Do not treat the label as proof of compliance. The EDPB Guidelines 4/2019 state that GDPR Article 25 does not require any specific technical and organizational measures, so a retention design cannot replace review of the applicable obligations.

Make the contract prohibit retention of document content. Require documentation of request logging, distinguishing session identifiers, timestamps, and latency from prompt or response content. Address debugging exceptions explicitly. Support tooling must not become an undocumented route for retaining content when an extraction exception occurs.

Require a signed Data Processing Agreement for personal data sent to the external processor, with a BAA where applicable. Get a current SOC 2 Type II report within 12 months, options for data residency, documentation of zero-retention, and a way to opt out of use as training data. Require the sub-processor list and its change notification policy too. The same package must include an incident response SLA, contractual right-to-delete, and audit-log access so that the buyer can assess handling obligations as well as output quality.

Build-vs-buy calculus and when the SLA negotiation is worth having at all

A managed extraction layer is not always the right purchase. Building makes more sense when the page volume stays high each month, the document types repeat and are under control, and an ML team is already committed to keeping it running. A data sovereignty or sector-compliance mandate that no vendor architecture can satisfy can also decide the question. A production build requires these operating conditions even after a successful prototype.

The prototype conceals work that appears after deployment. Every extracted value needs its exact page location for downstream validation, and that requirement is difficult to retrofit. The human review application can grow to outweigh the AI work. Frozen model code can break when the underlying model provider updates, and confidence calibration keeps taking work because a mostly correct pipeline can still fail silently.

Decide which layers to build, which to buy, and which team owns errors once the pipeline is live. For a managed layer, the preceding requirements are minimum due diligence regardless of contract size. A vendor that cannot meet them on the buyer’s documents is not the right fit. Building the extraction layer still calls for this negotiation, because the sub-components that are bought still need commitments.

Putting the checklist to work before any contract is executed

Send a written request for the accuracy methodology before discussing a percentage. Ask for per-field measurement against ground truth and for the names of the comparators used for strings, numbers, and dates. Make delivery of that definition a precondition for the test. Then provide the representative production sample and require results for every field, with standard and edge-case fields reported separately under that methodology.

Use the results to ask for the proposed SLA terms. Require field-level floors by field category and document type, a measurement period, a breach definition, a remediation mechanism, a claim window, and explicit exclusions. Review the exclusive-remedy language directly. Require the proposed carve-outs for consequential damages traceable to specific extraction errors to be addressed before execution.

Ask for the security package as part of the same talks. It must include the current SOC 2 Type II report, signed DPA, zero-retention architecture documentation, sub-processor list, and incident response SLA. Settle the other requirements that apply to data handling before signing. Confirm the pricing model as well, and explicitly ask whether payment can be tied to correctly extracted pages.

Treat each demand as a pass-or-incomplete decision. No written methodology, no test on your documents, or no current audit report means the buyer is being asked to absorb the accuracy risk entirely.