Processing documents at scale without inventing content

The failure that ends the project

You have a few hundred thousand documents and a date on a calendar. Contracts, claims packets, inspection reports, grant narratives, lab results — whatever the corpus, the ask is the same. Turn the pile into rows that a system downstream can act on.

The pilot on fifty documents looked good. Then someone in review opened a record, went to the cited page, and the value was not there. Not misread. Absent. The pipeline had produced a clean, plausible, correctly formatted effective date that appears nowhere in the source.

That single event costs more than every extraction error before it, because it changes the question being asked. Up to that moment the question was "how accurate is this." Afterward it is "how would we know." Once a reviewer catches one invented field, every field is suspect, including all the correct ones, and no accuracy figure you produce will retire the doubt. NIST's Generative AI Profile (NIST AI 600-1) names the class precisely: "Confabulation: The production of confidently stated but erroneous or false content (known colloquially as 'hallucinations' or 'fabrications') by which users may be misled or deceived."

The defense is not a better model. It is a system in which the invented value cannot reach the output, because every emitted field is required to carry a pointer into a source document and something mechanically checks that the pointer resolves. That constraint is the whole design. Everything below is what it takes to hold it at volume.

Extraction and generation are different contracts

Most document pipelines blur this distinction, and everything downstream inherits the blur.

Extraction means the output is a substring of the source. The system's job is to locate, not to compose. This is verifiable by construction: take the emitted string, take the cited offset, slice the document, compare. Its failure modes — wrong span, missing span, span that captures half the value — are all detectable without a human reading the page.

Generation means the output is new text: a summary, a rationale, a category label with its reasoning. It does not appear verbatim in the source, so no string comparison can check it. In "On Faithfulness and Factuality in Abstractive Summarization," Maynez, Narayan, Bohnet and McDonald (ACL 2020) found neural summarization models "are highly prone to hallucinate content that is unfaithful to the input document," with human annotators finding substantial hallucinated content in all model-generated summaries. The same paper reports that textual entailment measures correlate better with faithfulness than standard metrics — meaning the check for generated text is a semantic one, and semantic checks are slower and less certain than a byte comparison.

So the rule is: make every field extractive that can be extractive, and let normalization be deterministic code applied to the extracted span rather than something a model does silently in its head. Azure AI Document Intelligence is built this way — a field returns the content that was on the page alongside a normalized value, so an invoice date of "5/7/2022" surfaces as both the literal string and the ISO 8601 value "2022-05-07." If your record stores only "2022-05-07," you have thrown away the evidence and kept the conclusion.

Where generation is genuinely required, treat it as a separate class of field with its own governance: mark it as generated, attach the spans it was derived from, and never let it travel into a decision without them.

Layout is the substrate, and it fails quietly

Before any model sees your document, something decided what order the words are in. That decision is usually invisible and frequently wrong.

Reading order is the first trap. Microsoft's documentation states that content elements are "sorted by reading order that arranges semantically contiguous elements together, even if they cross line or column boundaries," and that where it is ambiguous "the service generally returns the content in a left-to-right, top-to-bottom order." On a two-column page with a sidebar, ambiguous is the normal case, and left-to-right stitches a left-column line to a right-column line into a sentence nobody wrote.

Page boundaries are the second. The same documentation notes that Document Intelligence "doesn't support reading order across page boundaries." A clause beginning at the bottom of one page and completing at the top of the next is, structurally, two unrelated fragments — and in contracts and regulations that is exactly where the qualifying language lives.

The third is furniture. Azure's layout model assigns paragraphs functional roles including page header, page footer, page number, title, section heading, and footnote. Use them: a footer repeating across hundreds of pages will otherwise enter your text stream hundreds of times and dominate any keyword index built over it. The instruction that follows from all three is the same — never split a document on raw character counts before you have seen how its layout parses. Split on the section and paragraph boundaries the parser reports, and carry the heading path, page number, and role onto every unit you create.

Tables break differently than prose

This is where correctness disappears most often and most silently, because a table that parsed wrongly still looks like a table.

Start with the distinction almost everyone misses: a visual table is not necessarily a data table. Microsoft's documentation is direct about it — layout tables "are extracted from tabular visual content in the document without considering the semantics of the content," and "some layout tables are designed purely for visual layout and don't always contain structured data." Getting structured data out of diverse visual layouts, it adds, "generally requires significant post processing." A grid on the page is a rendering decision; whether it holds a relation is a separate question your pipeline has to answer.

Then the structural cases: cells that span rows or columns, headers stacked two or three deep, stub-head cells in the top-left corner that describe both axes, description rows that break a table into sections mid-grid, and footnote markers attached to a number that change what the number means. Flattening any of these into a stream of values produces output that is arithmetically confident and semantically meaningless.

How hard this is shows up in the research record. In PubTables-1M (Smock, Pesala and Abraham, CVPR 2022) — nearly one million tables from scientific articles — the authors identified "oversegmentation" as a significant source of ground truth inconsistency in prior datasets and introduced a canonicalization procedure to remove it, reporting a significant increase in training performance and a more reliable estimate of model performance at evaluation. The transferable lesson is not about their model: a benchmark built by specialists needed an explicit canonicalization pass before its own table annotations agreed with each other. Two of your annotators labeling a merged header cell differently will surface later as "the model cannot do tables," and no model work will fix it.

Concretely: preserve row and column indices and span counts on every cell, carry the full header path down to each cell as metadata, keep footnote markers attached, and when you flatten a table into text for indexing, prepend the header row to each row-level unit so the numbers keep their meaning.

The span contract

Here is the single design decision that makes everything else tractable: define a per-field output record that cannot be constructed without provenance. Not a field that usually has a span. A record your code physically cannot write without one.

Every major API already hands you the primitives, and most pipelines discard them at the first transformation:

• Azure AI Document Intelligence — "Spans specify the logical position of each element in the overall reading order, with each span specifying a character offset and length into the top-level content string property." Alongside that, bounding regions give a 1-indexed page number and a bounding polygon, and extracted fields carry their own confidence value.
• Amazon Textract — per its response documentation, "Each block contains information about a detected item, where it's located, and the confidence that Amazon Textract has in the accuracy of the processing." The Geometry field returns a BoundingBox and a Polygon, and Relationships arrays link parent blocks to their children so a line resolves to its words and a table to its cells.
• Google Document AI — per its response guide, "The raw text is referred to in the textAnchor object which is indexed into the main text string with startIndex and endIndex," with a boundingPoly whose origin is the top-left corner of the page.
• Anthropic's citations API returns a char_location carrying cited_text, a 0-indexed start_char_index and an exclusive end_char_index. Its documentation states that because the API parses citations into a standardized format and extracts cited_text directly, "citations are guaranteed to contain valid pointers to the provided documents."

A minimum record, then: document identifier, content hash of the exact text the offsets refer to, page, character offset and length, bounding polygon, extracted literal, normalized value, extractor and model versions, confidence, timestamp. Nothing missing any of those gets written to the output table.

Two things this buys on day one. A reviewer can open the page and see the highlighted box, which is the single change that converts a skeptical subject-matter expert into a user. And when you change a model, you can re-run the corpus and diff by field rather than re-reviewing the whole thing by hand.

A schema guarantees shape, not truth

Constrained decoding is genuinely useful and routinely oversold. OpenAI's Structured Outputs documentation states the guarantee as "the model will always generate responses that adhere to your supplied JSON Schema." The same documentation, under best practices, notes that "Structured Outputs can still contain mistakes."

So a schema gets you parseable JSON, no missing required key, and no invalid enum value. It does not get you a correct value, a value that exists in the document, or a refusal when the field simply is not there. Schema adherence is a formatting property, not evidence.

Two schema fields carry most of the safety, and they are the two teams most often leave out. The first is a required source span on every extracted field, so an output without provenance is structurally invalid rather than merely disfavored. The second is an explicit representation for "not present in this document," distinct from null and distinct from an empty string. If the schema offers no way to say absent, you have built a system that must produce a value for every field on every document — and it will.

Verify the trace mechanically

Do not trust the offsets you were handed. Check them, on every document, in code. The check is cheap: take the emitted literal, take the cited offset and length, slice the canonical text, compare.

Four things go wrong here, and they are worth knowing before you debug them under deadline.

Normalization skew. The text you indexed was de-hyphenated, ligature-folded, or whitespace-collapsed somewhere in the pipeline, so the offsets refer to a different string than the one you are slicing. The fix is discipline: designate exactly one canonical text per document, hash it, store that hash on every record, and refuse to resolve a span whose hash does not match.

Character-unit mismatch. Azure's documentation notes that offsets and lengths are returned by default in user-perceived characters (grapheme clusters), with a query parameter to return them instead as Unicode code points or UTF-16 code units for different runtimes. A service consuming offsets produced for a different runtime will be silently wrong on any document containing an emoji, an accented character composed of combining marks, or a non-BMP glyph — and correct on everything else, which is what makes it hard to find.

Omitted zero coordinates. Google's documentation carries an explicit caution: when the API detects a coordinate value of 0, "that coordinate is omitted in the JSON response." Code that reads poly[0]["x"] gets a missing key, or worse, a language default that silently places the box somewhere it is not.

Pages that cannot carry spans at all. Anthropic's citations documentation notes that PDFs which are scans without extractable text are not citable. In any real mixed corpus some fraction of documents is in that category. Measure that fraction before you promise span coverage to a stakeholder, not after.

Every extraction should land in one of three states: verified (span resolves and the literal matches), unverified (no span, or the slice disagrees), and absent (the field is genuinely not in this document). Unverified is not a warning to log and move past. It is a routing decision — send it to a human queue or drop the field — and treating it as anything softer is how invented values reach production.

What a confidence score actually means

Vendor confidence describes the recognition, not your business rule. Textract's documentation states the operational implication plainly: "Depending on the scenario, detections with a low confidence might need visual confirmation by a human." That is confidence that the characters were read correctly and the block was typed correctly. It is not confidence that the dollar amount you pulled is the contract's total value rather than a line-item subtotal printed two inches above it.

Which means thresholds have to be calibrated on your own labeled data, per field, and not once per pipeline. The bar for a payment-driving amount is not the bar for a mailing address, and a single global cutoff produces the worst of both: too much review on fields that do not matter, too little on the ones that do.

Measure the failure that actually threatens the program, too. Precision and recall describe how often you were right about the fields you emitted; the number that answers the reviewer's real question is how often the system emitted a value with no support in the document. Our companion guide on precision, recall and false-extraction rate works through how those three interact and why the third one deserves its own gate.

The operational parts nobody budgets

Everything above is correctness. Scale adds a separate set of requirements, and they are the ones that tend to be discovered after a schedule has been committed.

Per-document isolation. One malformed PDF should fail one record, not the batch around it. This sounds obvious and is routinely violated by pipelines that batch at the wrong granularity.

Idempotency on content hash plus pipeline version. You will reprocess, more than once, and a re-run should be a cheap, safe, resumable operation rather than an event that needs a meeting.

Version stamps on every row. OCR version, model version, prompt version, schema version, extractor version. When quality moves, the first question is always "which version produced these rows," and you want that answered by a query rather than by archaeology.

A human queue, sized with arithmetic rather than optimism. Your review rate, times your document count, times fields per document, is a staffing plan — not a footnote. Run that multiplication with your own numbers before the pilot, because it is frequently the figure that decides whether a program survives its second year, and it is much harder to raise after an award than before one.

Per-field drift monitoring. Corpora change. A form revision arrives and precision falls on exactly one field while every aggregate metric stays flat. Without per-field measurement against a stable gold set, you find out from a user. For the wider set of ways ingestion degrades over time, our guide on why your data pipeline keeps breaking covers the upstream causes.

What finished looks like

A document system you can put in front of a hostile reviewer has four properties, and none of them are about the model.

Every field on every record points to a document, a page, and a character range that a reviewer can open and read in its original context. The system can say "this is not in the document," and does say it, often enough that people believe it. Extraction quality is a per-field number measured against a gold set and visible on a dashboard, rather than a shared impression formed during a demo. And a change that moves that number the wrong way fails a gate before anyone downstream sees a row.

If you are still deciding whether a document problem should be solved by extraction at all, our guide on RAG versus extraction covers the choice, model provenance and citation covers the traceability mechanics in more depth, and measuring an LLM hallucination rate covers how to put a number on the failure this article is built to prevent.

Sources: NIST AI 600-1 — Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile; Maynez, Narayan, Bohnet and McDonald, "On Faithfulness and Factuality in Abstractive Summarization" (ACL 2020); Smock, Pesala and Abraham, "PubTables-1M: Towards Comprehensive Table Extraction From Unstructured Documents" (CVPR 2022); Microsoft — Document Intelligence analyze document response; AWS — Amazon Textract text detection and document analysis response objects; Google Cloud — Document AI, handle the processing response; OpenAI — Structured Outputs; Anthropic — Citations. Analysis and framing by Precision AI Academy.

Common questions

What is the difference between extraction and generation in document processing? Extraction locates a substring that already exists in the source, so the output can be checked byte-for-byte against the cited character range. Generation composes new text — a summary, a rationale — which is not present verbatim and cannot be verified by string comparison. Make every field extractive that can be extractive, and run normalization as deterministic code over the extracted span.

How do I make every extracted field traceable to its source? Define a per-field record that cannot be written without provenance: document identifier, content hash, page, character offset and length, bounding polygon, extracted literal, normalized value, and extractor and model versions. Azure AI Document Intelligence returns spans as a character offset and length into the top-level content string plus bounding regions; Amazon Textract returns Geometry with a BoundingBox and Polygon per block; Google Document AI returns a textAnchor indexed with startIndex and endIndex.

Do JSON schemas and structured outputs prevent invented values? No. OpenAI's documentation states that Structured Outputs ensures the model "will always generate responses that adhere to your supplied JSON Schema," and the same page notes under best practices that "Structured Outputs can still contain mistakes." A schema removes missing keys and invalid enums; it does not make a value correct or present. Require a source span per field, and give the schema an explicit way to say the field is absent.

Why do tables break more often than plain text? Because a visual table is not necessarily a data table. Microsoft's documentation notes that layout tables are extracted from tabular visual content "without considering the semantics of the content," that some "are designed purely for visual layout and don't always contain structured data," and that structured extraction from diverse layouts "generally requires significant post processing." Spanning cells, stacked headers, and footnote markers add further ambiguity.