Precision, Recall and False-Extraction Rate, Explained

What the two numbers actually measure

Start with something concrete. A model reads 10,000 contract files and pulls out one field: the period-of-performance end date. When it finishes you have two separate questions, and they are not the same question.

Recall asks: of the end dates actually in those files, what fraction did the model find? Precision asks: of the end dates the model handed you, what fraction were right?

These definitions have been stable for more than thirty years. Nancy Chinchor's scoring paper for the Fourth Message Understanding Conference, MUC-4 Evaluation Metrics (1992), gives each a line: "Recall (REC) is the percentage of possible answers which were correct. Precision (PRE) is the percentage of actual answers given which were correct." MUC-4 borrowed both measures from information retrieval and extended them for extraction.

The important part is the two denominators, because they come from different places. Recall's denominator is possible — everything that was there to find. You cannot compute it from the model's output; somebody has to read the documents and write down the right answers first. Precision's denominator is actual — everything the model produced. That one is free.

That asymmetry explains most of what goes wrong when these numbers get quoted. Precision is cheap to estimate; recall requires ground truth, which is slow, expensive, and often the largest line item in an honest evaluation. A benchmark exact about precision and vague about recall may be showing the number that was easier to produce.

MUC-4 also fixed the bookkeeping underneath the two ratios, and modern tools still use its categories: correct (COR), partial (PAR), incorrect (INC), missing (MIS), spurious (SPU), non-committal (NON). Possible = COR + PAR + INC + MIS. Actual = COR + PAR + INC + SPU.

Why either number alone is meaningless

Both numbers are trivially gameable, in opposite directions, without touching the model.

Want precision of 100%? Have the system emit one extraction — the clearest, most boilerplate instance in the corpus — and stay silent everywhere else. One answer, one correct answer, perfect precision, useless system.

Want recall near 100%? Have it emit every date string in every document as a candidate. It will certainly capture the real ones. Precision falls through the floor.

Neither outcome requires a bad model. They are threshold settings. Any extraction system has a dial — a confidence cutoff, a decision to answer or abstain — and moving it trades one number against the other. MUC-4 stated the relationship in 1992 and it has not changed: "Normally as recall goes up, precision tends to go down and vice versa."

So a lone precision figure is not a claim about a system's quality. It is a claim about where somebody set a dial. The first question to ask about any single number is what the other one was at the same setting.

The third number: false extraction

"False-extraction rate" is not a standardized metric name, and you should be wary of anyone who presents it as though it were. But the idea behind it is standardized, and it is old.

MUC-4 separated two ways of being wrong that casual use lumps together. Incorrect means the source contained an answer and the system got the value wrong. Spurious means the source contained no answer for that field and the system produced one anyway. Both damage precision. Only the second is an invention.

MUC-4 gave the second its own metric: overgeneration, spurious divided by actual — "the percentage of the actual attempted fills that were spurious." The paper is explicit about why: raising recall by generating more fills was an obvious strategy, and "to avoid overpopulation of the template database by the message understanding systems, we introduced the measure of overgeneration."

That 1992 concern describes the modern one exactly. When a generative model reads a record set and writes a conclusion, the dangerous failure is not a date transcribed with the wrong year — a reviewer catches that by opening the record. It is the fluent, plausible sentence with no record behind it, because there is no record to open. Overgeneration is narrower than one-minus-precision, and narrower in the direction that matters.

The Attributable to Identified Sources (AIS) framework, published by Rashkin and colleagues in Computational Linguistics 49(4) in 2023, formalizes the same question for generated text: is a statement actually supported by the source it cites? Where a system's output is prose rather than a filled template, that is the shape the measurement has to take — every statement checked against the record it points to.

What a single F1 hides

F1 is the harmonic mean of precision and recall. MUC-4 used van Rijsbergen's F-measure for the same purpose, with the general form weighting recall against precision by a parameter β: β = 1.0 weights them equally, β = 0.5 makes recall half as important as precision, β = 2.0 makes it twice as important.

The harmonic mean punishes imbalance on purpose. As Chinchor notes, at β = 1.0 "a system which has recall of 50% and precision of 50% has a higher F-measure than a system which has recall of 20% and precision of 80%." That is intended — and it is the assumption you silently accept when you quote F1: that a missed item and an invented item cost the same. A missed clause in a contract review means someone reads that contract again; an invented clause means someone acts on a fact that does not exist. If those costs differ for your task, β should not be 1.

Two further cautions. D. M. W. Powers argues in "Evaluation: from precision, recall and F-measure to ROC, informedness, markedness and correlation" (arXiv:2010.16061) that recall, precision, F-measure and plain accuracy are all biased measures that should not be used without first identifying the chance or base-case level — how well a system would score by luck alone.

With more than one field or class, the averaging method changes the headline. The scikit-learn model evaluation documentation describes macro-averaging as "the mean of the binary metrics, giving equal weight to each class," and warns that "macro-averaging will over-emphasize the typically low performance on an infrequent class." Micro-averaging instead "gives each sample-class pair an equal contribution to the overall metric." The same predictions can yield materially different F1 values depending on which was used. Ask which.

The base rate that moves precision under you

Here is the effect that surprises people most, and it is pure arithmetic. Precision depends on how common the target is in the corpus. Recall does not.

A worked illustration with assumed numbers — made up to show the mechanism, not measurements of any product. Suppose a system finds 90% of true instances (recall = 90%) and produces a spurious extraction on 2% of the records that contain nothing to find. Hold both behaviors fixed and change only the corpus:

• Balanced corpus, 20% of 10,000 records contain the target. 2,000 true instances → 1,800 found. 8,000 empty records → 160 spurious. Precision = 1,800 / 1,960 ≈ 92%.
• Rare-event corpus, 2% of 10,000 records contain the target. 200 true instances → 180 found. 9,800 empty records → 196 spurious. Precision = 180 / 376 ≈ 48%.

Same system. Same recall. Same tendency to invent. Precision falls from 92% to 48% because the haystack changed. Notice too that in the second case the spurious extractions outnumber the correct ones — more noise than signal, while the recall number looks untouched.

Rare-event extraction is most of the interesting work: the unusual clause, the anomalous transaction, the one record in a thousand that matters. It is exactly where a precision figure from a curated, balanced evaluation set tells you least about your production corpus.

NIST addresses this directly. The AI Risk Management Framework (NIST AI 100-1) includes, under the MEASURE function, subcategory MEASURE 2.3: "AI system performance or assurance criteria are measured qualitatively or quantitatively and demonstrated for conditions similar to deployment setting(s)." Similar conditions include similar prevalence.

What counts as "correct" is a choice

Before any of these numbers exists, someone decided what a match is. That decision can move a headline several points without changing the model at all.

MUC-4 already had a partial-credit category, PAR, contributing fractional credit to both ratios. Entity-level evaluation today usually exposes the choice explicitly. The scheme popularized by SemEval-2013 Task 9 and implemented in open-source evaluators such as nervaluate distinguishes four ways of scoring the same output: strict ("exact boundary surface string match and entity type"), exact ("exact boundary match over the surface string, regardless of the type"), partial ("partial boundary match over the surface string, regardless of the type"), and type ("some overlap between the system tagged entity and the gold annotation is required"). Strict is the hardest; type is the most forgiving. A benchmark that does not say which it used has not told you what it measured.

The ground truth itself is the other soft spot. Chris Buckley and Ellen Voorhees showed in "Retrieval evaluation with incomplete information" (SIGIR 2004) that standard retrieval measures are not robust to substantially incomplete relevance judgments. Extraction inherits the problem in a misleading form: if the annotators missed instances, a model that finds those instances is scored as spurious. Precision is understated, recall overstated, and the error is invisible unless someone audits the gold set.

Reading a vendor benchmark critically

Eight questions. None are hostile — a vendor with real measurements answers each in a sentence.

• All three numbers, or one? Precision, recall and the spurious rate, at the same threshold setting. If only one is offered, ask for the others at that setting.
• What is the denominator? Not documents — instances. A precision of 90% measured over 200 instances has a 95% Wilson score interval of roughly 85% to 93% (Wilson, JASA, 1927). Over 20 instances the interval is wide enough to be uninformative.
• What was the prevalence? How often did the target appear in the test set, and how often does it appear in your corpus? See the arithmetic above.
• What was the matching rule? Strict, exact, partial or type. Partial credit is legitimate; undisclosed partial credit is not.
• Micro or macro averaged, over what classes? And what does the per-class table look like, not just the average.
• Who built the ground truth? Independently, or by reviewing the system's own output? The second is circular and inflates both numbers.
• What is the spurious rate on empty documents specifically? Test sets are usually built from documents that contain something; the records that contain nothing are where invention happens, and they are underrepresented.
• Was the evaluation data plausibly in the model's training corpus? A public benchmark that predates the model is not a held-out test set in any meaningful sense.

A note on scope, since we build these systems. Precision Federal (Precision Delivery Federal LLC, an Iowa limited liability company) builds small models that read through a body of data and produce a written conclusion with every statement traced back to the exact record it came from. Four things we will not do, listed because they are worth asking of anyone. We will not quote a precision or recall number for your data that we have not measured on your data. We will not report an F1 without the two numbers under it, the threshold that produced them, and the matching rule that defined "correct." We will not build ground truth by reviewing our own system's output and then score against it. And we will not present a metric threshold as satisfying an accreditation — an Authority to Operate is granted by an authorizing official against a control set, and no benchmark result substitutes for that work.

For the architecture side, see our guides on small reasoning models, RAG versus extraction, and agent evals.

Sources: Nancy Chinchor — MUC-4 Evaluation Metrics, Fourth Message Understanding Conference (1992); Rashkin et al. — Measuring Attribution in Natural Language Generation Models, Computational Linguistics 49(4), 2023; D. M. W. Powers — Evaluation: from precision, recall and F-measure to ROC, informedness, markedness and correlation (arXiv:2010.16061); scikit-learn — Metrics and scoring: micro and macro averaging; NIST — AI Risk Management Framework (AI 100-1), MEASURE function; Segura-Bedmar, Martínez & Herrero-Zazo — SemEval-2013 Task 9 (DDIExtraction 2013); nervaluate — entity-level evaluation schemas; Buckley & Voorhees — Retrieval evaluation with incomplete information, SIGIR 2004; E. B. Wilson — Probable Inference, the Law of Succession, and Statistical Inference, JASA 22(158), 1927. Worked examples are arithmetic from stated assumptions, not measurements of any product. Analysis and framing by Precision AI Academy.

Common questions

What is the difference between precision and recall? They share a numerator and differ in denominator. Recall is the share of the answers that were actually there that the system found; its denominator is everything that existed to be found. Precision is the share of the answers the system produced that were right; its denominator is the system's own output. MUC-4 states it as: recall is the percentage of possible answers which were correct, precision is the percentage of actual answers given which were correct.

What is a false-extraction rate? The share of a system's outputs that are inventions — statements produced where the source contained nothing to support them. It is not a standardized metric name, but it has a standardized ancestor: MUC-4's overgeneration, defined as spurious divided by actual. It is narrower than one-minus-precision, because precision also penalizes answers that exist in the source but were read wrong.

Why is a single F1 score not enough? F1 weights precision and recall equally, which assumes a missed item and an invented item cost the same. Often they do not, and the F-measure's β parameter exists precisely to reweight them. The same predictions can also yield different headline F1 values depending on whether the score is micro-averaged or macro-averaged across classes.

Why does precision drop when a vendor benchmark is applied to real data? Usually the base rate changed. Precision depends on how common the target is in the corpus, so a system with unchanged recall and an unchanged tendency to produce spurious output reports much lower precision on a corpus where the target is rare. NIST's AI RMF covers this under MEASURE 2.3, which calls for performance to be demonstrated for conditions similar to the deployment setting.