Document Processing Works When You Design the Escalation Path First

A system that is right 95% of the time and cannot tell you which 5% is worse than no system. Where the difficulty actually sits in invoice and contract automation, what a review queue needs to survive contact with an organisation, and the error rate nobody measures.

Document processing works when you design the escalation path first. The value is not in the share of documents the machine handles, it is in knowing, reliably, which ones it should not have handled. A system that is right 95% of the time and cannot tell you which 5% is worse than no system.

What this actually covers

Invoices, purchase orders, contracts, claims forms, remittance advice, delivery notes, policy documents, onboarding paperwork. Anything arriving in volume that a person currently reads and retypes into a system.

The work is extraction, pulling named fields out of an unstructured or semi-structured document, followed by validation, then either posting to a system of record or routing to a human. The extraction is the part vendors demonstrate. The validation and routing are the part that determines whether it works.

Where the difficulty really is

The two accuracy numbers, and which one you pay for

• Field-level F1. 0.903 to 0.943. Per-field accuracy across four architectures. The number that reaches a pitch deck.
• Document-level strict accuracy. 63.1% to 75.8%. Every field on the document correct at once. The number straight-through processing depends on.

Benchmark of 10,000 SEC filings (4,000 10-K, 4,000 10-Q, 2,000 8-K) across 11 GICS sectors, 25 extracted field types. The gap between the two columns is your exception queue.

Source: Kulkarni and Kulkarni, Benchmarking Multi-Agent LLM Architectures for Financial Document Processing, arXiv:2603.22651, March 2026.

Extraction from clean, typed, consistently laid out documents has been a solved problem for some time. Almost nothing arriving at a real organisation is clean, typed and consistent.

The hard cases, in roughly the order they cause trouble:

• Layout variation. Two hundred suppliers, two hundred invoice templates, several of which changed last year without notice.
• Scans of scans. A faxed copy of a printed copy of a PDF, arriving as an image with no text layer.
• Handwriting and annotation. The margin note that changes the payment terms.
• Attachments and amendments. The contract is forty pages, and the clause that matters is in a two-page amendment appended eight months later.
• Multi-document context. The invoice is only correct if it matches a purchase order and a goods-received note, which live in two other systems.
• Near-duplicates. The same invoice submitted twice with a different reference. Catching this is often worth more than the extraction.

A system that handles the first two and fails silently on the rest will look excellent in a pilot built from the first two.

Designing the escalation path first

What the exception queue needs before you buy anything

The queue is the part that decides the economics, and it is usually designed last.

• A measured document-level accuracy on your own documents. At least 200, including the difficult ones. Count documents that came out completely correct, not fields.
• A named owner for every exception type. An unrouted queue becomes a shared inbox, and a shared inbox becomes nobody's job.
• Confidence thresholds set per field, not per document. A wrong supplier name and a wrong line-item description do not carry the same cost.
• A cost per human touch. Loaded hourly rate divided by documents reviewed per hour. Without it you cannot size the saving.
• Volume testing at production scale. One benchmarked architecture lost 7 points of F1 between 1,000 and 100,000 documents a day.
• A feedback path from correction back into the system. If reviewer corrections do not change future behaviour, the queue never shrinks.

Nothing here is about model selection. Every item is about what happens to the documents the model could not finish.

Source: Thinklytics AI automation practice, 2026.

This is the part that separates the deployments that survive from the ones that get switched off after a quarter.

Every extracted field carries a confidence and every document carries a decision. Post automatically, hold for review, or reject. The thresholds are set by you and tuned against real consequences, not left at a default.

The review queue is a real queue with a real owner. Not an inbox. Somebody's job, with a service level, visible volume, and an ageing report. Most failures here are not technical: the queue was created, nobody was assigned to it, and within a month it held four thousand documents.

The system says why it escalated. Low confidence is not useful. Supplier not in master data, total does not match line items, no matching purchase order, amount exceeds threshold for automatic posting, each of those routes to a different person and a different fix.

Reviewer corrections feed back. The corrections are the most valuable training data the system will ever see, and in a surprising number of deployments they are discarded.

There is an audit trail per document. What was extracted, what was posted, who approved the exception, when. In finance and insurance this is not optional, and it is the thing that turns the machine did it into a defensible position.

What good looks like after twelve months

What OCR noise costs you, before the model sees anything

Identical extraction models run against clean reference text and against production OCR output.

• Forms, clean text
• Forms, OCR input
• Receipts (SROIE), clean text
• Receipts (SROIE), OCR input
• Receipts (CORD), clean text
• Receipts (CORD), OCR input

Source: Anvari and Athitsos, From Pixels to Pairs, arXiv:2609.17538, July 2026. Values are value-level F1, shown as percentages.

Not a percentage. Four things:

• The exception queue is stable or shrinking, and someone can name its owner.
• Cycle time per document is measured and was measured before the system arrived.
• The error rate on automatically posted documents is known, because a sample is audited. This is the number that matters and it is the one nobody collects.
• A named person is responsible for supplier template changes, because those will keep coming.

What to do before you buy anything

Take a hundred real documents, chosen by the people who process them today, specifically including the ones they find annoying, and establish three things.

What the current process actually costs. Minutes per document, error rate, rework. Measured now, before anyone can renegotiate the baseline.

How the shortlisted system performs on those hundred. Not a vendor demo set. Yours, including the bad ones.

What the exceptions have in common. This is frequently the most valuable output of the whole exercise. If sixty of your hundred problem invoices come from four suppliers, the answer may be a conversation with four suppliers rather than a software programme. That finding costs a fortnight and saves a great deal.

The part worth being clear about

Document processing is one of the more reliable applications in this category, with a measurable baseline and a measurable result. It is also routinely oversold, with automation rates quoted that describe a pilot on curated documents rather than the live queue.

Treat any quoted rate as a question rather than a fact: on whose documents, over what period, and what was the error rate on the ones it posted without asking.

Frequently asked questions

What does document processing automation actually cover?

Invoices, purchase orders, contracts, claims forms, remittance advice, delivery notes, policy documents, onboarding paperwork. Anything arriving in volume that a person currently reads and retypes into a system. The work is extraction, then validation, then either posting to a system of record or routing to a human.

Why is the escalation path more important than the extraction accuracy?

Because a system that is right 95% of the time and cannot tell you which 5% is wrong forces a human to check everything, which removes the saving. Value comes from knowing reliably which documents the machine should not have handled, so the rest can post without review.

What are the hard cases in document extraction?

Layout variation across hundreds of suppliers, scans of scans with no text layer, handwritten annotations that change terms, amendments appended months later, multi-document context where an invoice is only correct if it matches a purchase order and a goods-received note, and near-duplicates submitted twice with different references.

What does a review queue need to work?

A real owner rather than a shared inbox, a service level, visible volume and an ageing report. Escalation reasons that name the fault, such as supplier not in master data or total does not match line items, because each routes to a different person. Reviewer corrections fed back into the system. And an audit trail per document.

Which metric actually tells you the system is working?

The error rate on automatically posted documents, established by auditing a sample. Almost nobody collects it. Automation rate without that number tells you how much went through unchecked, not how much went through correctly.

What should we do before buying a document processing system?

Take a hundred real documents chosen by the people who process them today, including the annoying ones. Measure what the current process costs in minutes, errors and rework. Run the shortlist on those hundred. Then look at what the exceptions have in common.

What if most of our problem documents come from a few suppliers?

Then the answer may be a conversation with those suppliers rather than a software programme. If sixty of a hundred problem invoices come from four suppliers, fixing the input is cheaper and more durable than automating around it. That finding costs a fortnight and saves a great deal.