Reproducibility: Why It Matters More in Government Than Anywhere Else

A result is reproducible when someone who is not you, on a machine that is not yours, can run one command against the code and data you shipped and get the numbers you reported. Not similar numbers. The same numbers, or a documented and bounded spread around them.

In commercial software this is a good engineering habit that teams adopt when they have time. In federal work it is closer to a structural requirement, because four things happen to government systems that mostly do not happen to commercial ones. The work changes hands. Someone with statutory authority audits it. It has to be accredited before it is allowed to run. And a specific number you reported will be questioned long after the person who produced it has left the company.

This article covers why each of those four forces makes reproducibility load-bearing rather than optional, what reproducibility concretely requires, and how to build it in from the first commit instead of trying to reconstruct it during a transition.

What reproducible actually means

The word gets used loosely, so it helps to separate three levels. The ACM formalized this vocabulary in its Artifact Review and Badging policy, revised in 2020, and the distinctions are useful well outside academic publishing.

• Repeatable. The same team, using the same setup, gets the same result on a second run. This is the weakest bar, and a surprising number of projects fail it.
• Reproducible. A different team, using your artifacts (your code, your data, your environment specification), gets the same result. This is the level federal contracts need.
• Replicable. A different team, using its own independent artifacts, reaches the same conclusion. This is a research standard, not a delivery standard.
• Bitwise identical vs. statistically equivalent. A separate axis. Sometimes you can promise identical floating-point output. Often you can only promise that a rerun lands inside a stated interval. Both are defensible. Saying nothing about which one you mean is not.

When a contracting officer, an assessor, or a successor contractor says "reproduce it," they mean the second level, and they usually mean it with the fourth axis unstated. Your job is to state it.

Reason one: the work changes hands

Commercial products are usually maintained by the organization that built them, for as long as they matter. Federal work is not built that way. FAR Part 6 sets full and open competition as the default, which means the follow-on to the thing you built is competed, and it can be competed away from you. Option years end. A Phase I SBIR feeds a Phase II that another firm may support. Contracts get recompeted on a cycle that has nothing to do with whether your team still remembers how the model was trained.

When that happens, the government exercises its data rights and hands your deliverable to somebody else. For noncommercial work, the operative clauses are DFARS 252.227-7013 for technical data and DFARS 252.227-7014 for computer software and software documentation. Under Government Purpose Rights, the government may release your deliverable to other contractors for government purposes, and after the negotiated period (five years unless the parties agree otherwise) those rights typically convert to unlimited. SBIR data carries a longer protection period, set at 20 years from award by the SBIR/STTR Policy Directive, but "longer" is not "forever."

Here is the part teams miss. Those clauses give the government the right to receive a deliverable. They do not give the government a working system. If your repository builds only on the original author's laptop, with an undocumented Python version and a dataset that lives in a folder nobody archived, you have satisfied the letter of the delivery requirement and handed over something functionally worthless. The successor contractor spends the first four months of a transition rebuilding what you already built, the program office watches the schedule slip, and the record of that experience follows your firm.

Transition-in periods on federal contracts are commonly 30 to 90 days. That is the window in which a new team has to stand up your system with your people mostly gone. A reproducible build turns that from an archaeology project into a Tuesday.

Reason two: someone has the right to audit it

Commercial teams answer to customers. Federal contractors answer to customers, plus the agency Inspector General, plus the Government Accountability Office, plus the contracting officer, plus in some cases the Defense Contract Audit Agency. These are not metaphorical rights. FAR 52.215-2, the Audit and Records clause for negotiated contracts, gives the Comptroller General and the contracting officer access to your records, and FAR 4.703 requires contractors to make records available for three years after final payment on the contract, with specific retention periods by record type in FAR 4.705.

What auditors ask about AI systems is instructive. They rarely ask whether the model is good. They ask how the number was produced. GAO's own methodology guide on assessing data reliability (GAO-20-283G, published December 2019) turns on whether data are sufficiently reliable for the purpose at hand, and the way you demonstrate that is by showing the chain from raw source to reported figure. GAO's AI Accountability Framework (GAO-21-519SP, June 2021) organizes the same question into four principles, and the performance principle asks directly for documented methods, metrics, and results at both the component and the system level.

Agency-side obligations point the same direction. OMB Memorandum M-25-21, issued April 3, 2025, sets minimum risk management practices for high-impact AI use cases, including pre-deployment testing, an AI impact assessment, and ongoing monitoring. Agencies also maintain public AI use case inventories under the Advancing American AI Act, enacted as part of the FY2023 NDAA (Public Law 117-263). Every one of those obligations lands on the vendor as a request for evidence. If your evaluation is a notebook that ran once in March, you have nothing to send.

Reason three: it has to be accredited before it runs

Federal systems get an Authority to Operate before they touch production data. The process is the NIST Risk Management Framework (SP 800-37 Rev 2), and the control catalog is NIST SP 800-53 Rev 5. Read the configuration management family with reproducibility in mind and the overlap is close to total:

The SBOM requirement in particular is no longer discretionary in most federal software work. Executive Order 14028 (May 2021) directed the software bill of materials requirement, NTIA published the minimum elements in July 2021, and OMB memoranda M-22-18 and M-23-16 require producers to attest to secure development practices drawn from NIST SP 800-218, the Secure Software Development Framework. Two SSDF practices read like a reproducibility checklist written by a security office: PS.3.1 asks you to securely archive the files needed to reproduce a release, and PS.3.2 asks you to collect and safeguard provenance data.

The practical implication is worth sitting with. Teams often treat reproducibility work as overhead they cannot bill. It is not overhead. It is most of the configuration-management evidence the accreditation package already demands. Do it once and you have written a large fraction of your control responses as a side effect.

Reason four: defending a number a year later

Picture a sequence that is easy to end up in. Your proposal reported 94.2 percent precision at a 3 percent false-extraction rate on a document-processing task. You won. Fourteen months later, in an option-year review, a program manager says the fielded system is showing something closer to 88 percent and asks what happened.

There are three answers available to you. The first is "the original evaluation used a different test set," which invites the question of which one and why. The second is "we cannot reproduce the original run," which is the worst sentence in federal technical work. The third is "here is the manifest, here is the command, we reran it this morning on the original held-out set and got 94.1, and the gap in production comes from a shift in document mix that we can show you." Only the third answer keeps the conversation technical.

Two forces make this scenario common in AI work specifically. Models drift, and hosted model endpoints change underneath you. If you never pinned a dated model identifier and recorded it with the run, you cannot separate genuine data drift from a silent provider-side change from a bug you introduced. You are left arguing about a number with no way to interrogate it.

There is also a legal floor under all of this. Performance representations made in proposals and reports become part of the contract record, and the False Claims Act (31 U.S.C. 3729 through 3733) sets the outer boundary on knowingly false claims for payment. Almost no honest team ever gets near that boundary. But "we cannot reproduce the result we reported" is an uncomfortable place to be standing while anyone is asking questions.

The five artifacts reproducibility requires

Strip away the philosophy and reproducibility is five concrete things in your repository.

1. A pinned environment. Pin the base container image by its sha256 digest, not by a tag, because tags move. Pin every dependency with a lockfile that includes hashes: pip-compile --generate-hashes, poetry.lock, package-lock.json, or a fully exported conda environment. Record the CUDA and driver versions if you touch a GPU. A requirements file that says torch with no version is not a pinned environment; it is a wish.

2. Recorded randomness. Seed every generator you use, including the language runtime, the array library, the framework, and any data-shuffling utility. Set PYTHONHASHSEED. Then write the seed into the run record. The point is not that one seed is the true seed. The point is that the number you reported came from a run whose seed is known, and anyone can rerun that exact run.

3. A data manifest with hashes. For every input file: a SHA-256 digest, a byte size, a row count, the source URI or system of record, the date retrieved, and the handling marking. Splits are recorded as artifacts with their own hashes, never as "we did an 80/20 random split," which regenerates differently the moment a library changes its shuffling. The manifest is what lets someone confirm that they are evaluating on the same data you were, which is the single most common cause of two teams getting different numbers from the same code.

4. One command. From a clean clone on a clean machine, one command regenerates every number that appears in your report. Not a sequence of eleven notebooks run in a remembered order. One entry point, which internally does whatever it needs to.

5. A run record. Each execution writes a machine-readable record: the git commit SHA, whether the working tree was clean, the container digest, the config file used, the seeds, the model identifiers and versions, start and end timestamps, the hardware, and every metric produced. This file is the thing you attach to an audit response.

{
  "run_id": "eval-2026-08-11T14:22:05Z",
  "git_commit": "9f4c1ab3d7e02b58c1f9a4d6e8b0c72d1a5e3f90",
  "git_tree_clean": true,
  "container_digest": "sha256:1d3f...c0a7",
  "config": "configs/eval_holdout.yaml",
  "seeds": { "python": 20260811, "numpy": 20260811, "torch": 20260811 },
  "model": { "id": "vendor-model-2026-05-01", "temperature": 0 },
  "data": {
    "manifest": "data/MANIFEST.sha256",
    "holdout_rows": 4812,
    "holdout_sha256": "b41e...9cd2"
  },
  "metrics": { "precision": 0.9421, "recall": 0.8873, "false_extraction_rate": 0.0298 },
  "hardware": "1x A100 80GB, driver 550.54.15, CUDA 12.4",
  "wall_clock_seconds": 1187
}

The rule that ties it together: no number is typed into a document by a human. Every figure in the report, the briefing chart, and the proposal is read out of a generated record. If a number cannot be traced to a file that a script wrote, it does not go in.

The parts that are genuinely hard

It would be dishonest to present this as free. Three parts resist.

GPU nondeterminism. Many accelerated kernels use atomic operations whose accumulation order varies between runs, so identical inputs can yield outputs that differ in the last few bits. Deterministic algorithm modes exist in the major frameworks and they usually cost throughput, sometimes substantially. The honest position on most projects is statistical reproducibility: run the evaluation across several seeds, report the mean and the interval, and state which one you are promising. A single number with no spread invites exactly the challenge you cannot answer.

Hosted models. When your system calls a provider API, you do not control the weights. Pin the dated or versioned model identifier rather than an alias that silently rolls forward. Record it in the run record. And cache the raw responses, because a stored response set lets you recompute a metric from scratch even after the endpoint has changed. Temperature zero reduces variance; it is not a determinism guarantee across serving stacks.

Restricted data. You cannot archive controlled unclassified information into a general-purpose repository. CUI is defined and marked under 32 CFR Part 2002 and the categories in the NARA CUI Registry, and it stays inside the authorized boundary. The manifest still works: it carries the hashes, the row counts, the system of record, the retrieval procedure, and the marking, while the data itself stays where it belongs. Someone with the right access reproduces the result; someone without it can at least verify they are looking at the same data.

Building it in from day one

Retrofitting is the expensive path, and the reason is simple: by the time you need to reproduce a result, the environment that produced it is gone. The library version has moved, the developer's machine has been reimaged, the source system has been refreshed. Six steps, done in the first week, avoid all of that.

The transition package

When the work does change hands, this is the set of things that makes the handoff real. Assemble it continuously rather than at the end.

• Source code at a tagged commit, with the full history if data rights allow.
• Dependency lockfiles with hashes, and the container image referenced by digest.
• The SBOM in CycloneDX or SPDX, generated by the build rather than written by hand.
• The data manifest, including files you cannot ship, with retrieval instructions and markings.
• Seeds and configuration files for every reported run.
• Run records and stored raw model outputs for the evaluations cited in deliverables.
• The one command, plus a plain-language description of what it does and roughly how long it takes.
• A short list of known nondeterminism sources and the tolerance you are promising.

Common failure modes

The notebook that ran once. A result exists in cell output, the kernel has since restarted, and the cells were executed out of order. Nobody, including the author, can say what produced the number.

The floating dependency. An unpinned library publishes a new version, a default changes, and the same code now produces a different answer. Nothing in the repository records which version was in play when the reported result was produced.

The regenerated split. The test set was created by a random split without a fixed seed, or with a seed but a library whose shuffling implementation later changed. Two teams run identical code on identical raw data and disagree, and the disagreement takes a week to diagnose.

The hand-typed number. A figure in the report was transcribed from a terminal, then the pipeline was fixed and rerun, and the document was never updated. This one is common, quiet, and embarrassing in exactly the meeting where it surfaces.

The unarchived source system. The data came from a live database that has since been refreshed. There is no snapshot, and the original query returns different rows now. This is the failure with no remedy after the fact, which is why the manifest has to be written on ingest.

None of the five is exotic. Each one is a fifteen-minute decision at the start of a project that becomes an unrecoverable problem two years in. In commercial work you can usually absorb the loss and move on. In federal work the question comes from someone with the authority to ask it, on the record, and the answer becomes part of how your firm is judged the next time it competes.

Primary sources referenced: FAR Part 6; FAR 4.703 and 4.705; FAR 52.215-2; DFARS 252.227-7013 and 252.227-7014; SBIR/STTR Policy Directive (data rights period); NIST SP 800-37 Rev 2; NIST SP 800-53 Rev 5 (CM and SR families); NIST SP 800-218 (SSDF); Executive Order 14028; NTIA SBOM minimum elements (July 2021); OMB M-22-18, M-23-16, and M-25-21; Advancing American AI Act (Pub. L. 117-263); GAO-20-283G; GAO-21-519SP; 32 CFR Part 2002; 31 U.S.C. 3729 to 3733; ACM Artifact Review and Badging v1.1 (2020). Verify clause applicability against your own contract before acting on any of it.