In This Article
Key Takeaways
- On August 6, 2026, Ishan Patel, Sahil Sen, Elias Lumer and Vamse Kumar Subbiah posted The Bitter Lesson of Tool Calling (arXiv:2608.06370) comparing two tool interfaces across 14 language models on a 309-entry BFCL v4 subset.
- Programmatic tool calling, where tools are typed Python stubs the model invokes by writing a script, matched or exceeded native JSON tool calling on 11 of 14 models, and on 13 of 14 under parallel fan-out.
- GPT-5.6-Sol and GPT-5.6-Terra each gained 10.6 percentage points over their own JSON baseline. All five Anthropic models tested matched or beat baseline.
- The three models that lost failed on script syntax, not reasoning: they emitted literal escape sequences instead of real newlines and the subprocess refused to run.
- Costs move in both directions. Code used 1.5x the input tokens on short chains, fewer tokens above roughly 26 parallel calls, and about half the wall-clock time on chaining for 13 of 14 models.
Every team that builds agents eventually argues about the same design question. Should the model call a tool by emitting a JSON object that a framework dispatches, or by writing a few lines of code that call the tool as an ordinary function? Both work. Almost nobody has run the comparison on the same models, the same tasks, and the same scoring.
On August 6, 2026, four researchers posted one to arXiv. The Bitter Lesson of Tool Calling (arXiv:2608.06370), by Ishan Patel, Sahil Sen, Elias Lumer and Vamse Kumar Subbiah, tests both interfaces across 14 models and publishes the per-model table. The short version: code matched or beat JSON on 11 of 14 models, the three losses had nothing to do with reasoning ability, and the size of the win tracked how recently a model shipped.
What they actually ran
The benchmark is BFCL v4, the Berkeley Function Calling Leaderboard, which scores whether a model picks the right function and fills its arguments correctly. The authors used a 309-entry subset spanning eight task categories, against 14 models released between November 2024 and July 2026: five from Anthropic (Claude Haiku 4.5, Sonnet 4.5, Sonnet 4.6, Opus 4.8 and Sonnet 5) and nine from OpenAI (GPT-4o, GPT-4.1, GPT-5-nano, GPT-5, GPT-5.4-mini, GPT-5.4, and the three GPT-5.6 tiers Luna, Sol and Terra).
The baseline condition is what most frameworks do today. The model receives, in the paper's words, “function schemas formatted as JSON tool definitions,” and emits one structured JSON object per call through the provider's tool-calling endpoint. Every call is a round trip through the model.
The test condition, programmatic tool calling, gives the model “typed Python stubs compiled from the benchmark's function schemas.” The model writes a script that imports those stubs, and the agent loop “executes it in a shell subprocess, producing tool-call results without additional inference turns.”
That final clause is the entire mechanism. Three dependent calls cost three inference turns in the JSON condition, because each result has to come back to the model before it can compose the next call. In the code condition they cost one, because the dependency lives inside the script where a variable can hold the intermediate value.
The headline number
Across the full subset, programmatic tool calling matched or exceeded the JSON baseline on 11 of the 14 models. Here is the per-model comparison as the paper reports it.
Overall accuracy on the 309-entry BFCL v4 subset (paper, Table 2)
| Model | JSON baseline | Programmatic |
|---|---|---|
| Claude Haiku 4.5 | 81.9% | 85.8% |
| Claude Sonnet 4.5 | 86.4% | 87.7% |
| Claude Sonnet 4.6 | 80.9% | 87.4% |
| Claude Opus 4.8 | 84.8% | 84.8% |
| Claude Sonnet 5 | 84.5% | 86.1% |
| GPT-4o | 81.9% | 55.0% |
| GPT-4.1 | 81.9% | 62.1% |
| GPT-5-nano | 66.7% | 68.3% |
| GPT-5 | 71.5% | 76.1% |
| GPT-5.4-mini | 79.3% | 55.0% |
| GPT-5.4 | 79.3% | 81.9% |
| GPT-5.6-Luna | 76.1% | 80.3% |
| GPT-5.6-Sol | 72.2% | 82.8% |
| GPT-5.6-Terra | 73.5% | 84.1% |
The largest gains sit at the newest end. The paper states that “GPT-5.6-Sol and GPT-5.6-Terra each achieve an absolute improvement of 10.6% over their own JSON tool calling baseline.” One small inconsistency to flag, since it will show up in secondary coverage: the abstract and the body both say 10.6%, while the concluding paragraph says 10.7%. The table supports 10.6.
The three losses were a formatting bug
GPT-4o, GPT-4.1 and GPT-5.4-mini all scored materially worse writing code than emitting JSON. The reason is worth sitting with, because it is not the reason most people would guess. The paper describes it directly: the models “produce code with literal \n escape sequences in multiline scripts rather than real newlines, causing the subprocess to fail with a syntax error on any entry requiring more than a single-line script.”
Those models did not fail to understand the tools. They failed to produce a file Python would parse. Everything downstream then scored zero. The authors' own summary of the pattern is that the remaining gap correlates “with model generation rather than model family,” and the table reads that way: the five Anthropic models and the three newest GPT-5.6 tiers all held or improved, while two 2024-era models and one mini tier collapsed on syntax.
Fan-out and schema flooding
Two of the three ablations probe conditions that break agents in production.
The parallelism ablation (n=32) asks for many independent calls at once, sweeping fan-out from 7 to 48 with extra probes at N of 60, 70, 72, 75 and 100. Programmatic tool calling matched or outperformed the baseline on 13 of 14 models. The sharpest single result: Claude Sonnet 5's JSON baseline drops to 0% at N=100, while the code condition maintains 100% enumeration accuracy. A loop does not get bored at the ninetieth item. Emitting one hundred separate well-formed JSON objects in a single response is a different kind of task.
The context rot ablation (n=31) floods the prompt with decoy function schemas, 128 in total, and compares three conditions: a filtered prompt with only the relevant schemas, the flooded prompt, and a filesystem-discovery approach where the agent has to go find its tools. The paper reports that “the mean accuracy change from filtered to flood is an absolute −2.3% for JSON tool calling and +5.5% for programmatic tool calling,” while “the filesystem-based internal condition degraded by an absolute 32.0% on average under flooding, where every model declined.”
Read that third number carefully before generalizing from it. Making the model discover its own tools on disk was, in this setup, far more fragile than either of the two interfaces the paper is actually comparing.
What it costs
The chaining ablation (n=52) runs sequential multi-hop calls, chains of 2 to 20, where the output of one function has to become the argument of the next. Programmatic tool calling “completes chaining entries in roughly half the wall-clock time of baseline for 13 of 14 models, with per-entry latency ratios ranging from 0.32 to 0.96 of baseline.” That follows from removing inference turns rather than from any cleverness.
It is not free. On that same ablation, the code condition “uses 1.5× the input tokens of JSON tool calling,” because the typed stub definitions ride along in the prompt. On parallel fan-out the arithmetic inverts: the crossover sits near N of about 26, and past it the code condition is cheaper. At N=30 the paper reports 3,559 tokens for JSON against 3,380 for code; at N=48, 5,097 against 3,535.
So the honest summary is that code costs more input tokens on short chains, fewer on wide fan-out, and less wall-clock time in nearly every case. If the token overhead is what worries you, note that stub definitions are a stable prefix, which is exactly the shape prompt caching is good at.
What the paper does not show
The authors are unusually direct about their own limits, and the caveats matter more than the headline.
- BFCL v4's functions are echo-return stubs rather than live APIs, so the study measures “argument serialization accuracy” rather than whether real services behave.
- Ablation samples run from n=31 to n=52, which the paper says produces “wide confidence intervals.” Small per-model differences in those tables are noise.
- The paper cites a recent audit finding “20% evaluator-human misalignment” in BFCL's LLM-judge mode. Treat single-digit gaps between adjacent models as unresolved.
- It measures accuracy, tokens and latency. It is not a security paper. The execution step is described as a shell subprocess, and the operational question of running model-written code near real systems sits outside the scope.
One more, since the title invites the question: we found no citation to Rich Sutton's essay anywhere in the text. Read the title as a nod, not a claim about scaling.
Why it matters
The following is our analysis, not reported fact. Four things a practitioner can do something with.
The tool interface is a performance lever, not a style preference. Ten points of accuracy and half the latency is the size of win teams normally chase through weeks of prompt rewriting. If your agent chains calls or fans out over a list, this is worth an afternoon of measurement.
The result is generation-dependent, so it expires. A study run on 2024 models would have concluded the opposite, because the older models could not emit a clean script. That means the finding does not transfer by citation. It transfers by re-running it on your models and your tasks, which is a small, well-defined experiment: same tools, same task set, both interfaces, measured. Our guide to how serious teams test agents covers how to build one.
This is not an argument against MCP. Nothing in the paper tests it. MCP governs how tools are advertised and reached; programmatic tool calling changes how a model invokes tools it has already been given. The paper's related work cites CodeAct, Hugging Face's smolagents, and Cloudflare's Agents platform code execution mode as prior systems pointing the same direction, which suggests the two layers compose rather than compete.
Test the boring failure first. If code-style tool calling underperforms for you, check whether your model is emitting real newlines and a parseable file before concluding anything about its reasoning. In this study that single formatting defect accounted for all three losses. It is the kind of thing a two-line assertion in your test rig catches permanently, and the kind of thing that otherwise gets misdiagnosed as a model that “can't do agents.”
Measure it on your own tasks
Benchmarks tell you what happened to someone else's task set. Our guide covers building an eval rig that answers the question for yours.
Read the agent evals guideSources: Ishan Patel, Sahil Sen, Elias Lumer, Vamse Kumar Subbiah, “The Bitter Lesson of Tool Calling,” arXiv:2608.06370 (submitted August 6, 2026, cs.CL) and the full HTML text, source of every figure and quotation above; Berkeley Function Calling Leaderboard (BFCL) V4 for benchmark background. Analysis and framing by Precision AI Academy.
Common questions
What is programmatic tool calling? In this study, tools are exposed as typed Python stubs compiled from the benchmark's function schemas. The model writes a script that imports and calls them, and the agent loop runs that script in a subprocess, so a chain of calls resolves without extra inference turns.
Should I switch my agent to it? The paper does not say that, and neither do we. It reports a result on echo-return stubs, on one benchmark subset, with small ablation samples. It is a strong reason to run the comparison yourself, on live tools, before changing anything.
Which models handled it best? All five Anthropic models tested matched or exceeded their JSON baseline, as did the three GPT-5.6 tiers. The largest single gains were GPT-5.6-Sol and GPT-5.6-Terra at 10.6 points each.
Does this replace JSON tool calling? No. Three models did worse with code, the token overhead on short chains is real, and JSON tool calling remains the default path in every major SDK. The finding is that code is a viable and sometimes better alternative for models new enough to write clean scripts.