How to Evaluate an AI Vendor's Accuracy Claim

Suppose a vendor tells you their system reads your forms at 94% accuracy. You are not a statistician. You have forty minutes, a program to run, and no way to rerun their experiment. What do you ask?

This article is that list. None of the questions require math you cannot do in your head, and all are answerable by a vendor who has genuinely measured what they claim. The goal is not to catch anyone lying; most vendors are not. It is to learn whether their number predicts what happens when the system meets your data, which is harder than whether the number is true.

What a bare percentage is missing

"94% accurate" is a sentence fragment. A complete performance claim has six parts, and a slide rarely carries more than one:

• The metric. Accuracy on what unit: per character, field, page, document, case?
• The dataset. Which documents, from where, collected when, labeled by whom.
• The size. How many items were scored.
• The procedure. How it was split, how many runs, what was tuned and on what.
• The uncertainty. The spread across runs, or an interval around the estimate.
• The comparison. What the alternative scores under the same procedure.

You are entitled to all six. The NIST AI Risk Management Framework (NIST AI 100-1) says as much in its MEASURE function, which asks that performance be demonstrated under conditions resembling the deployment setting rather than a laboratory one. OMB memorandum M-25-22 (April 2025) tells federal agencies to build performance expectations and testing into AI contracts instead of accepting vendor assertions.

Question 1: What data was this measured on?

Start here, because everything downstream depends on it.

Where did it come from? Public benchmark, synthetic generation, the vendor's own customers, or a customer like you? Public benchmarks are the most convenient and least informative: one that has been on the internet for three years has very likely appeared in the training data of any model built after it. That is contamination, and it inflates scores in a way nobody can audit. Synthetic data is usually too clean, rarely reproducing the smudged fax, the margin note, or the form version retired in 2019 that still arrives in your intake queue.

Who produced the labels, and how much did they disagree? Accuracy is scored against an answer key, and people make the answer key. If two annotators labeled the same 200 documents, what fraction did they agree on? If they agreed 91% of the time, a system reported at 94% is scored against a ruler with visible wobble. A vendor who has never computed inter-annotator agreement has not examined their own ground truth.

How old is it? A 2023 snapshot may not describe 2026 traffic.

Question 2: How was the data split?

A model must be measured on data it did not learn from. The subtle failures are in how the separation was drawn.

Random splits leak when records are related. If one contract file produced 40 pages scattered randomly between training and test, the model saw 30 pages of that contract before being asked about the other 10. Split by entity instead, so every page of a contract lands on one side of the line. Ask: by row, or by document, case, claimant, or facility?

Time-ordered data needs a time-ordered split. Anything predictive, from demand forecasting to fraud scoring, must train on the past and test on the future. A random split lets the model learn from next quarter to predict last quarter, a capability you will not have in production.

The test set gets reused. A team measures, adjusts a prompt, measures again, moves a threshold, measures again. Twenty iterations later the test set has functioned as a training set through the engineer's hands, and the score is optimistic by an unknown margin. Ask how many times it has been scored.

Question 3: How many runs, and how wide is the spread?

Results vary across random seeds, data orderings, prompt phrasings, and the batching behavior of the serving stack. A single run hides all of it. Reimers and Gurevych (EMNLP 2017) found that comparing single scores between sequence-tagging models led to different conclusions than comparing score distributions. Henderson et al. (AAAI 2018) showed that different seeds of the same reinforcement learning algorithm could be arranged to look like two algorithms with a clear winner. Dodge et al. (EMNLP 2019) argued for reporting the search budget spent finding a result, since 50 attempts and 3 attempts do not produce comparable numbers.

You do not need to cite any of that in a meeting. You need one question: how many times did you run this, and what was the range? If the answer is "once," read the number as "somewhere in this vicinity, probably." If the answer is "five runs, 91% to 95%," you know something real, and you know a competitor at 93% sits inside their noise.

Even from one run, the sample size tells you a great deal. A 95% confidence interval around a proportion follows from the count and the total:

Same 90% claim, different test-set sizes

Test set	Correct	Reported	95% interval (Wilson)
100 items	90	90%	roughly 83% to 94%
1,000 items	900	90%	roughly 88% to 92%
10,000 items	9,000	90%	roughly 89.4% to 90.6%

Same headline, completely different evidence. If your acceptance threshold is 88%, the 100-item result cannot tell you whether the system clears it.

When the interval crosses zero

Comparisons carry their own uncertainty, wider than either number alone. Suppose Vendor A scores 92% and Vendor B scores 89%, each on 200 documents. A three-point win looks decisive on a slide. Run the interval on the difference and it spans roughly minus 3 to plus 9 points. Zero sits inside it, so the data are equally consistent with B being slightly better.

That is what "crossing zero" means, and it is the most useful statistical idea a buyer can carry. When the interval around a difference includes zero, the honest verdict is a tie, decided on other grounds: price, delivery risk, security posture, how the team behaved during the evaluation.

The consequence is scoping. If a source-selection decision will rest on measured accuracy, size the test set for the gap you care about; if you can only afford 100 documents, accuracy screens out gross failures and the award turns on something else.

Question 4: What is it compared against?

A number alone means nothing. "94% accurate" is interesting only relative to what happens without the system. Three baselines are worth demanding.

The trivial baseline. What does the dumbest possible approach score? For classification, predicting the majority class every time. If 3% of invoices are fraudulent, a system that flags nothing is 97% accurate, so a vendor quoting 96% on fraud detection is quoting a result worse than doing nothing. That sounds like a joke; it is one of the most common real problems in imbalanced-class procurement.

The incumbent baseline. What does your current process score, measured the same way on the same data? Few organizations know their own error rate, which is why the vendor's number so often floats free. Measuring it converts a percentage into a business case.

The honest simple baseline. Regular expressions, keyword rules, a small classical model. A vendor comparing a heavily tuned system against a baseline written in an afternoon has produced a marketing artifact. The fair question: if your team had spent the same two weeks tuning the baseline, what would it score? Good engineers find that interesting. It is a bad sign when it lands as an insult.

Question 5: Does the metric match the decision?

Accuracy gets quoted because everybody understands it. It is frequently the wrong metric.

Errors are rarely symmetric. Missing a real safety defect and flagging a clean part each count as one error under accuracy, and they are not equal in consequence. You want the split: how many real cases were missed, and how many false alarms raised, at the operating point you would use. That is precision and recall, and a vendor who cannot give both at a stated threshold has not built an evaluation you can use.

Aggregate scores hide per-item math. A 40-field form extracted at 95% accuracy per field is fully correct about 13% of the time, if errors are roughly independent. At 99% per field it is about 67%. When every field must be right before a reviewer can stop looking, per-field accuracy describes a system that almost never finishes the job, in language that sounds like it almost always does.

Summary curves hide the operating point. A high area-under-the-curve figure averages behavior across every threshold, including ones nobody would set. If you will run at a threshold that keeps false alarms under five per day, ask for recall there. Ask for throughput from the same run too: two points of accuracy bought with eleven extra seconds per document is a bad trade under a service-level commitment.

Question 6: Can you reproduce it?

The strongest evidence is a result you regenerate yourself. The weakest is a slide.

Reproducing a claim requires six things pinned down: the model version and date, the decoding settings, the full prompt or configuration, the retrieval index in play, the test data, and the scoring script. Change one and the number moves. Providers ship model updates continuously, so a result measured in February against a since-updated model describes a system that no longer exists. Ask when it was taken and against which pinned version. Expect small residual variation even then, because serving stacks batch requests and floating-point arithmetic on accelerators is not associative. Treat any margin below that variation as noise.

The practical form: can we run it ourselves, on our data, this month? A few hundred of your own documents, scored by your own people, run against every finalist under identical conditions, teaches more than any deck. Federal buyers can often fund that kind of paid bake-off below the simplified acquisition threshold, and the results are usable in source selection as long as the method appears in the solicitation before proposals arrive, consistent with FAR 15.305.

Question 7: What is the failure regime?

This is the question I would keep if I could keep only one.

Every real system fails on an identifiable slice of inputs. Handwriting. Multi-column layouts. Scans below 200 dpi. Documents over 60 pages. Names with diacritics. Anything the training data underrepresented. A team that has measured its system knows that slice by heart and describes it in specific, slightly weary detail. A team that says "it performs consistently across document types" has either not looked or decided not to say.

Ask for twenty errors. Not a summary: the actual failed items, with the model output and the correct answer side by side. Reading them tells you what no confusion matrix conveys, which is whether the failures are the boring kind a review step catches, or the confident, plausible, wrong kind that slips past a reviewer moving fast. For generative systems the second kind is the whole risk, which is why measuring unsupported outputs is a separate exercise from measuring accuracy.

Ask for the worst slice too. If the overall figure is 94% but the lowest slice scores 61%, and your intake is 40% that slice, the headline was never about you.

Question 8: What did you not report?

Selective reporting is rarely dishonest and almost always present. Four patterns to name out loud:

• Best-of-N configurations. Twelve prompts were tried; the best is on the slide. With twelve attempts, something looks good by chance alone. Ask how many were evaluated.
• Dropped cases. "We excluded documents where OCR failed." Those are documents, and they still arrive in production. Ask what fraction of the corpus was excluded and why.
• The favorable subset. The number covers the three document types that work well; the other five are absent.
• Post-hoc metric choice. Four metrics were measured and the best-looking one is reported. Ask which was chosen before the experiment ran.

The friendly way to surface all four at once: "What did you try that did not work?" It separates teams who did real engineering from teams who assembled a demo. The first group has a long, interesting answer, and that answer is the best predictor I know of whether a pilot survives production, a bar most pilots do not clear.

Turning the answer into contract language

A number in a deck is marketing. A number in your contract is a requirement. Closing that gap is ordinary contracting work.

Write the performance claim into inspection and acceptance terms under FAR Part 46: the metric, the unit, the threshold, the test population, the sample size, and who scores it. "Field-level extraction accuracy of at least 92%, measured on a government-selected random sample of 500 production documents per quarter, scored by government adjudicators against the published data dictionary" is a requirement. "94% accurate" is not.

Three details make the clause work. Name who picks the sample, because a vendor-selected sample is a self-graded exam. Size the remedy to a real quality problem rather than to noise. Require re-measurement on a schedule, because model versions change underneath you and a threshold met at award is not a threshold met in month fourteen.

The one-page script

Bring this list. You do not need to understand the statistics behind the questions to use the answers.

1. How many items were in the test set?
2. Where did that data come from, and when?
3. Who made the answer key, and how often did two labelers disagree?
4. How was it split, and how many times has that test set been scored?
5. How many runs, and what was the range?
6. What is the baseline, who built it, and how hard did they try?
7. Precision and recall at the threshold we would run? At our unit of work.
8. Which model version, date, prompt, and index? And can we rerun this ourselves on our data.
9. What is your worst slice, and what does it score? Show me twenty errors.
10. How many configurations did you evaluate, and what did you exclude?

Ask in a spirit of curiosity rather than cross-examination; vendors who have done the work enjoy these questions, and their enthusiasm is itself a signal. And if a vendor answers all ten cleanly while quoting a lower number than a competitor's unexamined one, the lower number is usually the better bet. A measured 87% you understand beats an unmeasured 96% you do not.

Sources and notes: NIST AI Risk Management Framework (NIST AI 100-1, January 2023), MEASURE function; OMB M-25-22, "Driving Efficient Acquisition of Artificial Intelligence in Government" (April 2025); FAR 15.305 and FAR Part 46. Reporting practice: Reimers and Gurevych, EMNLP 2017; Henderson et al., AAAI 2018; Dodge et al., EMNLP 2019. Intervals are Wilson score intervals at 95% confidence from the stated counts; the 1,500-per-arm figure is a two-proportion sample-size calculation at 80% power for 89% versus 92%.

Common questions

What if confidentiality prevents sharing the test data? That is legitimate and common, and it does not stop a vendor from giving you the size of the set, the labeling procedure, the split method, the number of runs, the spread, and the worst slice. If confidentiality is offered as a reason to withhold the procedure, the procedure is the problem.

We have no labeled data. Where do we start? Label 200 items by hand, with two people, and measure how often they agree before you look at any system. That produces a test set, a measurement of your own process, and a clearer definition of what "correct" means. Most teams find that definition was fuzzier than they thought, which is worth knowing before a vendor optimizes against it. The same set feeds straight into the build-versus-buy decision.