How to Measure an LLM Hallucination Rate

Decide what counts as a hallucination

"Hallucination rate" sounds like a specification, the way clock speed is a specification. It is not. It is the output of a procedure you designed, and two competent teams measuring the same model on the same corpus can publish numbers that differ several-fold without either being dishonest.

The first split is between faithfulness and factuality. Faithfulness asks whether the output is supported by the source the system was given; factuality asks whether it is true of the world. Maynez et al. (ACL 2020) found abstractive summarization models "highly prone to hallucinate content that is unfaithful to the input document," and separated intrinsic hallucination, which contradicts the source, from extrinsic, which introduces content not present in it. Extrinsic content is not automatically wrong — a model can add something correct the document never said.

That is where measurement programs go astray. If your system reads a defined body of records, a true-but-unsupported sentence is still a defect: the reader was promised a conclusion drawn from those records and cannot check that sentence against them. Decide explicitly whether unsupported-but-true counts. Either answer is defensible; leaving it undecided means each annotator resolves it differently and your rate inherits the disagreement.

Note what this rules out. TruthfulQA (Lin, Hilton and Evans, ACL 2022) — 817 questions across 38 categories — is closed-book: the best model was truthful on 58% of questions against 94% for humans, with the largest models generally the least truthful. Useful about world knowledge; silent on whether a model stays inside the documents you hand it.

The unit you count changes the number

Same outputs, two rates. Consider a 400-word answer containing roughly forty distinct assertions, one of which the source does not support.

• Response-level: the fraction of outputs containing at least one unsupported statement. This answer is a failure. Rate: 100% on this sample of one.
• Claim-level: the fraction of individual assertions that are unsupported. Rate: 2.5%.

Both are honest, and they are not comparable; a headline figure with no stated unit is uninterpretable. The response-level rate is closer to what a user experiences, since a reader who finds one bad sentence distrusts the paragraph. The claim-level rate is the more sensitive instrument for tracking change.

The careful claim-level design is FActScore (Min et al., EMNLP 2023), which breaks a generation into atomic facts and computes the percentage supported by a reliable knowledge source; the authors report an automated estimator with less than a 2% error rate against their human evaluation — for their setting and their knowledge source. Decomposition is itself a modeling step: something decides where one claim ends and the next begins, and that judgment carries its own error into your denominator.

Report both rates, plus the average number of claims per response. Without the third number, a claim-level rate improves by writing longer answers.

Who decides: humans, entailment models, LLM judges

Something has to read an output against its source and rule. Three families, three failure profiles.

Human annotation is the reference standard and the expensive one. It works only with a written guideline, and the guideline is the real design: what counts as supported, how paraphrase is treated, what to do with a partially supported sentence. Consider a published rubric rather than a new one — Rashkin et al. define the Attributable to Identified Sources (AIS) framework and its two-stage annotation pipeline. Then measure inter-annotator agreement and publish it: if two trained humans agree only 70% of the time, no automatic metric built on their labels can be trusted past that ceiling.

Entailment-based metrics ask a natural language inference model whether the source entails the generated sentence. Maynez et al. found textual entailment measures correlate better with faithfulness than standard metrics — with the corollary that ROUGE and similar overlap scores are not hallucination measurements at all. The engineering detail comes from SummaC (Laban et al., TACL 2022): earlier work found NLI models uncompetitive at inconsistency detection, which the authors traced to a granularity mismatch — NLI training data is sentence-level, inconsistency detection is document-level. Segmenting the document into sentences and aggregating pairwise scores fixed it; their SummaC-Conv reached 74.4% balanced accuracy on a six-dataset benchmark, five points above prior work. Treat that as a working detector, not an oracle: right three times in four is a useful filter and a poor instrument when the rate you are chasing is in low single digits. TRUE (Honovich et al., NAACL 2022) found across 11 standardized datasets that large-scale NLI and question generation-and-answering approaches give strong and complementary results — running both often beats tuning one.

LLM-as-judge is now the default because it is easy. Zheng et al. (NeurIPS 2023 Datasets and Benchmarks Track) found strong LLM judges can match human preferences with over 80% agreement, and documented the accompanying biases: position, verbosity, self-enhancement, and limited reasoning ability. Self-enhancement is the one that quietly corrupts a hallucination measurement, because the convenient judge is usually a sibling of the model under test.

The rule covering all three: measure the judge. Hand-label a few hundred examples from your own data, then report the judge's agreement with those labels next to the rate it produced. A rate from an unvalidated judge is a statement about the judge.

The denominator and the abstention loophole

A system that answers fewer questions hallucinates less. That is the easiest way to produce an impressive number by accident, and it is why a rate reported without coverage is incomplete. Report a small table instead: answered and supported, answered and unsupported, abstained where an answer existed, abstained where none did. The last cell is a success, not a miss.

Then settle the retrieval question in writing. If the retriever never surfaced the passage and the model answered from memory, is that a hallucination or a retrieval failure? Both choices are legitimate — one measures the end-to-end system, the other isolates the generator. Deciding case by case during annotation is not. We covered the architectural split in RAG vs extraction.

Finally, the rate belongs to a full configuration — model version, prompt, decoding temperature, chunk size, retrieval depth, reranker — not to a model name. Change the chunking and the number changes. Version-pin the evaluation harness the way you version-pin code, and record the configuration with every result.

How wide is that number, really

This is the step most often skipped. Suppose you sample 100 outputs and find 3 unsupported ones. The point estimate is 3%; the 95% Wilson score interval is approximately 1.0% to 8.5%. A competing system measured at 6 out of 100 has an interval of roughly 2.8% to 12.5%. Those overlap across most of their range, so the evidence does not support calling either system better. Raising the sample to 30 out of 1,000 keeps the estimate at 3% and narrows the interval to about 2.1% to 4.3%.

Use the Wilson interval rather than the textbook normal approximation. Brown, Cai and DasGupta, "Interval Estimation for a Binomial Proportion" (Statistical Science, 16(2), 2001, 101–133), describe the coverage properties of the standard Wald interval as chaotic and recommend Wilson or the equal-tailed Jeffreys interval for small samples, Agresti-Coull for larger ones. Hallucination rates are exactly the regime Wald handles worst: small proportions, modest samples.

One design choice buys more precision than a bigger sample: run both systems on the same items and compare per item. A paired comparison removes the variance contributed by item difficulty, usually the largest source of noise here. Comparing two independently drawn rates throws that away.

Building an evaluation set that stays honest

Two kinds of evaluation set do two different jobs, and conflating them is a common error.

A random sample of real traffic estimates the rate you actually experience in production; it is the only set from which "our hallucination rate is X" means anything. An adversarial or stratified set — long documents, conflicting sources, near-duplicate records, questions with no answer in the corpus — finds failure modes efficiently, but a rate computed on it is not a rate, because the sampling was deliberately unrepresentative. Build both; label which is which.

Include unanswerable items deliberately. SQuAD 2.0 (Rajpurkar, Jia and Liang, 2018) added over 50,000 unanswerable questions precisely so systems had to determine when no supported answer exists and abstain. If every question in your set has an answer, you have not tested the behavior that matters most under review.

Keep the set private. Public benchmarks can end up in training corpora, and a number produced on a leaked benchmark measures memorization. A held-out set from your own documents, never published and never used for tuning, is the only instrument that stays calibrated.

Measuring inside an accreditation boundary

If the data is controlled, the evaluation harness is part of the system, not a tool beside it. The DoD Cloud Computing Security Requirements Guide defines the Impact Levels governing where a category of data may be processed, and DoD Provisional Authorizations at IL4, IL5 and IL6 are issued by the DISA Authorizing Official on the basis of FedRAMP plus additional DoD requirements, per DISA's Cloud Authorization Process briefing (June 2024). We cover the levels in DoD Impact Levels explained.

The consequences are concrete. The judge model, the annotation interface, the stored outputs and the labels all inherit the impact level of the data they touch. Sending controlled text to an outside API to be graded is not an option, which usually means a small entailment model running locally — validated against human labels on site, since you cannot import someone else's agreement statistics. Disconnected environments tighten this further; we walked through those in air-gapped AI deployment.

What survives an audit is not a low number but a documented procedure. The NIST AI Risk Management Framework (NIST AI 100-1, January 2023) organizes its MEASURE function around evaluating systems against trustworthiness characteristics and providing a traceable basis for decisions. A reviewer who can reproduce how you got 3% will accept 3%. A reviewer handed 3% with no method will not, and should not.

What we will not do. We are Precision Federal, and our own work sits on the traced side of this problem — small models that read through a body of data and produce a written conclusion, with every statement traced back to the exact record it came from. Our edges, stated plainly. We will not quote a hallucination rate without naming four things: the evaluation set, the unit counted, the judge, and the judge's measured agreement with human labels on that data. We will not carry a published leaderboard figure over to someone else's corpus and call it a prediction. We will not report a point estimate without its interval, and we will say when a sample is too small to separate two systems. We do not certify or accredit models, we are not a third-party assessment organization, and we do not issue authorizations — we build the model layer to live inside a boundary someone else accredits. And we will not tell you a rate can be driven to zero. Anything that composes free text can compose an unsupported sentence; the engineering question is whether every sentence is bound to a record a reviewer can open.

Sources: Maynez et al., On Faithfulness and Factuality in Abstractive Summarization (ACL 2020); Lin, Hilton & Evans, TruthfulQA (ACL 2022); Min et al., FActScore (EMNLP 2023); Rashkin et al., Measuring Attribution in Natural Language Generation Models (AIS); Laban et al., SummaC (TACL 2022); Honovich et al., TRUE: Re-evaluating Factual Consistency Evaluation (NAACL 2022); Zheng et al., Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena (NeurIPS 2023); Rajpurkar, Jia & Liang, Know What You Don't Know (SQuAD 2.0, 2018); Brown, Cai & DasGupta, Interval Estimation for a Binomial Proportion (Statistical Science, 2001); DoD Cloud Computing Security Requirements Guide; DISA, DoD Cloud Authorization Process (June 2024); NIST AI 100-1, AI Risk Management Framework. Confidence intervals computed with the Wilson score method. Analysis and framing by Precision AI Academy.

Common questions

What is an LLM hallucination rate? It is the fraction of a model's outputs, or of the individual claims inside those outputs, that are not supported by the source the model was given. It is not a fixed property of a model. It is the result of a measurement procedure, and the procedure — the evaluation set, the unit counted, the judge, the sample size — determines the number as much as the model does.

What is the difference between faithfulness and factuality? Faithfulness asks whether the output is supported by the source document that was provided. Factuality asks whether the output is true of the world. Maynez et al. (ACL 2020) separate the two and distinguish intrinsic hallucination, which contradicts the source, from extrinsic hallucination, which is not present in the source and may still be true. For a system that reads a specific body of records, faithfulness is usually the measurement that matters.

Can an automatic metric replace human labeling? Only after the automatic metric has itself been measured against human labels on your data. Entailment-based detectors are useful but imperfect: SummaC-Conv reached 74.4% balanced accuracy on a six-dataset benchmark (Laban et al., TACL 2022). LLM judges carry documented position, verbosity and self-enhancement biases (Zheng et al., NeurIPS 2023). Report the judge's agreement with human labels alongside the rate.

How many samples do I need to measure a hallucination rate? More than most teams use. Three unsupported outputs in a sample of 100 is a 3% point estimate with a 95% Wilson score interval of roughly 1.0% to 8.5%. A system measured at 6% on the same sample size has an interval of roughly 2.8% to 12.5%, which overlaps heavily, so the two cannot be called different. Thirty in a thousand narrows the interval to roughly 2.1% to 4.3%.