AI benchmark metrics for document extraction in 2026

Quick Answer: An AI benchmark for document extraction needs labelled test documents and Google's precision, recall and F1 metrics⁴. Microsoft's field confidence estimates predictions rather than test-set accuracy; custom table, row and cell confidence starts with API 2024-11-30 GA⁵. Record the dataset, scoring method and API version before comparing results.

Summary in a mind map

AI benchmark metrics for document extraction in 2026
│
├─ Define the tested set
│   ├─ Hold back blind documents
│   ├─ Freeze labels and scoring rules
│   └─ Name the business document mix
│
├─ Read Google's metrics
│   ├─ Precision: matched predictions
│   ├─ Recall: found annotations
│   └─ F1: balance of precision and recall
│
├─ Read Microsoft's signal
│   ├─ Field confidence is per prediction
│   └─ Custom table confidence begins in 2024-11-30 GA
│
└─ Report limits
    ├─ Put the API version beside each result
    └─ Do not infer accuracy on unseen documents

Benchmark accuracy versus local accuracy

An AI benchmark score describes a particular evaluation, not every document a business will process. NIST distinguishes accuracy on the questions in a benchmark from generalised accuracy across a broader universe of similar questions¹. For document extraction, that distinction becomes practical as soon as the test documents differ from the invoices, forms or correspondence that arrive in production. The score should travel with a description of the tested material. Context matters.

Suppose a purchasing team sees a strong result on clean, consistently formatted invoices. Its own incoming invoices may have other layouts and scanned pages. That difference is a reason to run another test, not a reason to call the published score false. A report can show both results, with each dataset named. Both remain bounded. The first measures the published set; the second measures a held-out business sample. Neither should be relabelled as universal accuracy.

A fair comparison holds the task steady. Give each system the same field definitions, annotation policy and document set. Separate extraction errors from disagreements about the correct label. Otherwise a change in the scoring rule can look like a change in model quality. This is a test design recommendation grounded in NIST's distinction between the stated task and its implementation^1,3.

Protect the blind test set

NIST's AI Technology Evaluation overview describes blind data in a sequestered testbed as a way to mitigate training and test contamination and support objective assessment². The principle matters even in a small procurement trial. If a supplier tunes against every document that later appears in the final score, the report no longer represents a blind evaluation. Keep one set for development and another that remains unseen until the comparison is fixed.

NIST defines evaluation cheating as exploiting a gap between what a task is meant to measure and how the task is implemented, in a way that undermines validity³. A document test can have such a gap without any dramatic misconduct. If the intended question is whether a system extracts a payable amount but the scoring script accepts a nearby subtotal, the script measures a different task. Write the target field and acceptance rule before collecting scores.

The procurement team should retain the version of the annotation guide used by human labelers. A score without that guide leaves a reviewer unable to tell whether a missed value was model error or a disputed ground truth. This is especially relevant when several figures on a document look plausible. Ambiguity needs a rule. Resolve cases in the annotation policy, then apply the same policy to all products.

Google's labelled document test

Google Document AI compares predicted entities with annotations in test documents⁴. Its evaluation reports precision, recall and F1 for labels. This is an evaluation against labelled examples, so the result depends on the content and annotations in the test set. A business should name the processor, document group and label set next to the numbers. A single headline score hides whether the fields the team actually needs were represented.

The reported metrics answer related but different questions. Precision is true positives divided by true positives plus false positives; recall is true positives divided by true positives plus false negatives⁴. Google defines F1 as the harmonic mean of precision and recall⁴. A team that cares strongly about omissions can look at recall for a critical field rather than using F1 alone. A team troubled by invented fields can examine precision. The preferred balance follows the cost of each error in the business workflow.

For an accounts-payable example, the team may care separately about supplier name, invoice number and total. Show each tested label if the evaluation supplies it. A high aggregate result could conceal a weaker field that drives payment approval. The accounts-payable automation guide describes the broader process; the benchmark here measures the extraction step.

Microsoft's field confidence

Microsoft Document Intelligence describes field confidence as an estimated probability, between 0 and 1, that a prediction is correct⁵. This is attached to a prediction. It can inform a decision to accept the field automatically or send it for human review. That use is operational, but it does not make confidence identical to an aggregate benchmark score. One is a signal about an output; the other summarises performance against labelled examples.

A practical trial can log each extracted field, its confidence value, the human-confirmed value and the action taken. This lets the team see whether its proposed review threshold works on its own documents. The threshold itself is a business decision. The cited Microsoft guidance does not specify a universal acceptance level for every organisation or field⁵.

The confidence signal is useful only if the team knows which output it covers. A report saying "confidence 0.9" without naming the field, model and API version cannot show what was estimated. If several fields are essential to one approval, review the decision rule as a whole rather than treating one field's score as approval of the document.

The Microsoft API version limit

Microsoft states that confidence scores for custom-model tables, rows and cells start with API version 2024-11-30 GA⁵. This is a specific version boundary, not a claim that all prior versions lack every confidence field. A trial that compares table extraction should therefore record the API version actually used on each run. If one system exposes row confidence and another configuration does not, the result sheet should say so directly.

The version entry belongs beside the model and result date. A vendor demonstration may use a different endpoint or version from the organisation's deployed application. The comparison then needs a repeat test under the intended configuration before a purchasing decision. Service quotas and throughput limits need separate, current documentation.

A result sheet that can be checked

A useful result sheet keeps the conditions with each number. The denominator matters. State the dataset source, document count, label definitions, held-out status, processor or model, API version, metric name and scoring rule. Then present the value and any field-level breakdown. These details let another reviewer understand which result is a test-set score and which is a prediction-level confidence value^1,4,5.

The team should also describe exclusions and unresolved labels. A case removed because its annotation was ambiguous should be visible in the test record, not silently omitted after seeing a model's answer. This is an application of NIST's concern with evaluation validity³. It makes the denominator of each reported score intelligible.

A separate trial log can record the human review decisions made on low-confidence fields. That log tests a workflow, rather than a model. For a wider selection method, see the agent opportunity score, AI agents for smaller firms, and the distinction between agents and automation platforms. None of those pages replaces a held-out extraction test.

What this result cannot establish

The cited sources do not supply Google or Microsoft quota limits, request-size limits, regional endpoint limits or a general throughput comparison. These limits cannot be inferred from precision, F1 or field confidence. A buyer needs a separate current service-limits source before making a volume promise.

Nor does a benchmark on one held-out sample prove how every future file will perform. NIST's distinction between test-set and generalised accuracy is the reason to state the claim narrowly¹. A production rollout can monitor newly arriving document types and review errors rather than recycling the pilot result as a permanent guarantee.

What to do next

Define the extraction fields, freeze a blind sample and specify the scoring rules before comparing systems. Keep Google's precision, recall and F1 beside the labelled dataset description⁴. Keep Microsoft's field confidence beside the prediction and API version, especially for custom tables, rows and cells⁵. Approve a result only when its test conditions can be reproduced.