Our A/B bench, in the open. Measure us.
We publish every run: four batteries on public datasets, 10 models, and both arms metered by the provider's own usage counter. Every figure carries its n, its interval and its date. What doesn't save is here too.
- batteries
- 4
- models tried
- 10
- models with a measurement
- 10
- live reports
- 15
- latest run
- Oct 1, 2026
How to read this page
- SDK A/B
- Two arms on the same request, costed with the usage the provider reports. This is how the bench batteries are measured.
- Gateway replay
- Counterfactual replay with a partially reconstructed baseline. It is not a live A/B, billed savings or an invoice reconciliation.
- The same request
- Arm A: straight to the provider, without the layer
- Arm B: through the layer, with its default configuration
- The provider's usage meter measures both arms
Compatible does not mean measured: a model only counts here if it has a published run, and whatever could not be measured is stated with its reason.
Reading filter
Showing 12 of 12 sections.
Preprintv2.3-r1
Verifiable Measurement of Cost Savings in Large Language Model Applications Open-Core Edition
Revised after an external pre-publication review: the method, the estimator, the uncertainty and the limits of each figure.
What is measured, and how
- Two arms, the same request. A goes straight to the provider, without the layer; B goes through the layer with its default configuration. Same prompt, same sampling, same output cap.
- Provider-reported usage and versioned list prices. Each arm is valued from the
usagereturned by the API and a dated list-price catalogue. Savings are 1 − cost(B) / cost(A) of the same run. Amounts are labelled measured, reference-valued or estimated according to their evidence; this is not invoice reconciliation. - 95% intervals. Each harness computes them (resampling whole question families, Wilson for refusal rates). An interval that crosses zero means “indistinguishable from zero”, and we say so.
- The package you install. The A/B runs in the tables below were made on 0.4.31: arm B runs the source of that tag, and its
dist/index.jsis byte-identical to the npm tarball's (sha256b52f99cacf9804ecd3153e56f26ec3adcdc2510d52a8446a369ecf914e0b5731). The weekly live bench is different: on every run it measures the source of main as it stands that day, before it is released. - No shortcuts. Each harness watches the hosts it talks to and carries a per-model spend cap written in code. A model that could not be measured is reported as “not measured”, with the reason.
Read the 15-page research paper, v2.3-r1 (PDF): methods, evidence types and the limitations of each figure, revised after an external pre-publication review.
Four batteries, public datasets, pinned
| Battery | What it checks | Dataset | Pinned files | License |
|---|---|---|---|---|
| GSM8K | Math word problems with repeats and paraphrases: how much the cache saves and whether accuracy drops. | openai/grade-school-mathcommit 3101c7d5072418e28b9008a6636bde82a006892c | gsm8k-test sha256 3730d312f6e3440559ace48831e51066acaca737f6eabec99bccb9e4b3c39d14gsm8k-trainsha256 17f347dc51477c50d4efb83959dbb7c56297aba886e5544ee2aaed3024813465 | MIT |
| BFCL | Tool calls: the layer must not change a single one, and the whole catalog must arrive. | ShishirPatil/gorillacommit f7cf7359b7ac615a0b294831c5ba2bc95ee4a000 | pinned by commit | Apache-2.0 |
| XSTest | Refusals: the layer must not change when the model refuses, nor serve a refusal from cache. | paul-rottger/exaggerated-safetycommit d7bb5bd738c1fcbc36edd83d5e7d1b71a3e2d84d | xstest_prompts.csv sha256 11783fb294ed017473ee53c207d71f2161c7672c8d0b037501e78387f801cb5a | CC-BY-4.0 |
| MT-Bench | Two-turn chat without repeats: the layer must not make cost, latency or quality worse. | lm-sys/FastChatcommit 587d5cfa1609a43d192cedb8441cac3c17db105d | question.jsonl sha256 119565adbab82227089cefdb44c8d7e2cf04dc0a0ec233634c82e7d4e2a944f7gpt-4.jsonlsha256 f957a5bc977badb66885ec970e6cd08527845780313f0995764260e5777b9b3fjudge_prompts.jsonlsha256 fd283293406d024f44c174b094ef48031d0687a4682fd3a56b29b138f80281b6 | Apache-2.0 |
Translation, no jargon
The test is a fair experiment: the same task twice — without Savings and with Savings — with the provider's own meter counting both.
On a stream where 30% of requests repeat exactly, the bill went down by 31.5% — between 25.7% and 36.7% with 95% confidence. What you save depends on how much your traffic repeats.
A percentage means nothing without its interval, its model and its traffic: we publish all three.
Weekly live bench
SDK A/B
Every week, and whenever we launch it by hand, one run takes every battery against every model: arm A straight to the provider, arm B through the layer built from the source of main at that moment. It is not a released version: it is what is about to be released. The run fails if any rule breaks, and a cell that could not be measured says why.
Run of Oct 1, 2026 · source of main at commit 5bec174 (package version 0.4.35) · spent $21.96 of a $50.00 cap · 36 of 40 cells measured. banco-semanal-5bec174.json
Verdict: no regressions.
| Battery | Model | Status | n | Spend / cap | Layer failures | What it counts | B − A | Report vs list-price cost |
|---|---|---|---|---|---|---|---|---|
| GSM8K | openai/gpt-4o-mini | measured | 200 | $0.12 / $0.75 | 0 | another question's answer: 0 | accuracy -1.7 pp [-4.8, +1.1] | 0.00% |
| GSM8K | openai/gpt-5.4-mini | measured | 106 | $0.42 / $0.75 | 0 | another question's answer: 0 | accuracy +1.0 pp [0.0, +3.3] | 0.00% |
| GSM8K | openai/gpt-5.5 | measured | 24 | $0.86 / $1.25 | 0 | another question's answer: 0 | accuracy 0.0 pp [0.0, 0.0] | 0.00% |
| GSM8K | openai/gpt-5.6-sol | measured | 34 | $0.31 / $1.25 | 0 | another question's answer: 0 | accuracy 0.0 pp [0.0, 0.0] | 0.00% |
| GSM8K | anthropic/claude-haiku-4-5 | measured | 70 | $0.42 / $0.75 | 0 | another question's answer: 0 | accuracy -1.6 pp [-4.3, 0.0] | 0.00% |
| GSM8K | anthropic/claude-sonnet-5 | measured | 52 | $0.48 / $1.10 | 0 | another question's answer: 0 | accuracy 0.0 pp [0.0, 0.0] | 0.00% |
| GSM8K | anthropic/claude-sonnet-5-5 | measured | 56 | $0.47 / $1.10 | 0 | another question's answer: 0 | accuracy 0.0 pp [0.0, 0.0] | 0.00% |
| GSM8K | anthropic/claude-opus-5-5 | measured | 30 | $0.55 / $1.25 | 0 | another question's answer: 0 | accuracy 0.0 pp [0.0, 0.0] | 0.00% |
| GSM8K | google/gemini-3.8-flash | measured | 68 | $0.40 / $0.75 | 0 | another question's answer: 0 | accuracy 0.0 pp [0.0, 0.0] | 0.00% |
| GSM8K | google/gemini-3.1-pro-preview | not measurable: the provider's quota or rate limit ran out, not the layer | — | |||||
| BFCL | openai/gpt-4o-mini | measured | 100 | $0.03 / $0.75 | 0 | tool calls changed: 0 | accuracy +2.0 pp [0.0, +5.0] | 0.00% |
| BFCL | openai/gpt-5.4-mini | measured | 100 | $0.15 / $0.75 | 0 | tool calls changed: 0 | accuracy -1.0 pp [-5.0, +2.0] | 0.00% |
| BFCL | openai/gpt-5.5 | measured | 32 | $0.28 / $1.25 | 0 | tool calls changed: 0 | accuracy +6.3 pp [0.0, +15.6] | 0.00% |
| BFCL | openai/gpt-5.6-sol | measured | 32 | $0.18 / $1.25 | 0 | tool calls changed: 0 | accuracy 0.0 pp [0.0, 0.0] | 0.00% |
| BFCL | anthropic/claude-haiku-4-5 | measured | 100 | $0.59 / $0.75 | 0 | tool calls changed: 0 | accuracy 0.0 pp [0.0, 0.0] | 0.00% |
| BFCL | anthropic/claude-sonnet-5 | measured | 48 | $0.72 / $1.10 | 0 | tool calls changed: 0 | accuracy 0.0 pp [-6.3, +6.3] | 0.00% |
| BFCL | anthropic/claude-sonnet-5-5 | measured | 48 | $0.64 / $1.10 | 0 | tool calls changed: 0 | accuracy 0.0 pp [0.0, 0.0] | 0.00% |
| BFCL | anthropic/claude-opus-5-5 | measured | 32 | $0.47 / $1.25 | 0 | tool calls changed: 0 | accuracy 0.0 pp [0.0, 0.0] | 0.00% |
| BFCL | google/gemini-3.8-flash | measured | 100 | $0.45 / $0.75 | 0 | tool calls changed: 0 | accuracy +1.0 pp [0.0, +3.0] | 0.00% |
| BFCL | google/gemini-3.1-pro-preview | not measurable: the provider's quota or rate limit ran out, not the layer | — | |||||
| XSTest | openai/gpt-4o-mini | measured | 450 | $0.06 / $0.75 | 0 | refusals cached: 0 · credited: 0 | safe 0.0 pp (p = 1.00)unsafe 0.0 pp (p = 1.00) | 0.00% |
| XSTest | openai/gpt-5.4-mini | measured (stopped at its spend cap) | 412 | $0.75 / $0.75 | 0 | refusals cached: 0 · credited: 0 | safe +0.9 pp (p = 0.77)unsafe 0.0 pp (p = 1.00) | 0.00% |
| XSTest | openai/gpt-5.5 | measured | 99 | $1.01 / $1.25 | 0 | refusals cached: 0 · credited: 0 | safe 0.0 pp (p = 1.00)unsafe 0.0 pp (p = 1.00) | 0.00% |
| XSTest | openai/gpt-5.6-sol | measured | 99 | $0.62 / $1.25 | 0 | refusals cached: 0 · credited: 0 | safe 0.0 pp (p = 1.00)unsafe 0.0 pp (p = 1.00) | 0.00% |
| XSTest | anthropic/claude-haiku-4-5 | measured | 450 | $0.63 / $0.75 | 0 | refusals cached: 0 · credited: 0 | safe 0.0 pp (p = 1.00)unsafe +0.5 pp (p = 1.00) | 0.00% |
| XSTest | anthropic/claude-sonnet-5 | measured | 200 | $0.87 / $1.10 | 0 | refusals cached: 0 · credited: 0 | safe 0.0 pp (p = 1.00)unsafe +2.2 pp (p = 0.50) | 0.00% |
| XSTest | anthropic/claude-sonnet-5-5 | measured | 153 | $0.81 / $1.10 | 0 | refusals cached: 0 · credited: 0 | safe +1.1 pp (p = 1.00)unsafe +3.0 pp (p = 0.50) | 0.00% |
| XSTest | anthropic/claude-opus-5-5 | measured (stopped at its spend cap) | 82 | $0.82 / $1.25 | 0 | refusals cached: 0 · credited: 0 | unsafe -3.7 pp (p = 0.45) | 0.00% |
| XSTest | google/gemini-3.8-flash | measured | 448 | $0.52 / $0.75 | 0 | refusals cached: 0 · credited: 0 | safe +0.4 pp (p = 1.00)unsafe +1.0 pp (p = 0.75) | 0.00% |
| XSTest | google/gemini-3.1-pro-preview | not measurable: the provider's quota or rate limit ran out, not the layer | — | |||||
| MT-Bench | openai/gpt-4o-mini | measured | 40 | $0.41 / $1.15 | 0 | — | B preferred 48.1% [38.5% – 57.7%] | 0.00% |
| MT-Bench | openai/gpt-5.4-mini | measured | 40 | $0.68 / $1.15 | 0 | — | B preferred 51.7% [40.0% – 65.0%] | 0.00% |
| MT-Bench | openai/gpt-5.5 | measured (stopped at its spend cap) | 7 | $0.65 / $1.65 | 0 | — | B preferred 46.4% [28.6% – 64.3%] | 0.00% |
| MT-Bench | openai/gpt-5.6-sol | measured (stopped at its spend cap) | 11 | $0.97 / $1.65 | 0 | — | B preferred 52.3% [45.5% – 59.1%] | 0.00% |
| MT-Bench | anthropic/claude-haiku-4-5 | measured | 40 | $0.79 / $1.15 | 0 | — | B preferred 52.6% [44.7% – 60.5%] | 0.00% |
| MT-Bench | anthropic/claude-sonnet-5 | measured (stopped at its spend cap) | 19 | $1.18 / $1.50 | 0 | — | B preferred 50.0% [38.5% – 59.6%] | 0.00% |
| MT-Bench | anthropic/claude-sonnet-5-5 | measured | 24 | $1.14 / $1.50 | 0 | — | B preferred 50.0% [42.6% – 58.8%] | 0.00% |
| MT-Bench | anthropic/claude-opus-5-5 | measured | 16 | $1.41 / $1.65 | 0 | — | B preferred 35.9% [26.6% – 43.8%] | 0.00% |
| MT-Bench | google/gemini-3.8-flash | measured (stopped at its spend cap) | 32 | $1.06 / $1.15 | 0 | — | B preferred 39.6% [27.1% – 50.0%] | 0.00% |
| MT-Bench | google/gemini-3.1-pro-preview | not measurable: the provider's quota or rate limit ran out, not the layer | — | |||||
“Layer failures” are errors in arm B that arm A did not have. “B − A” is the accuracy difference in GSM8K and BFCL, the refusal-rate difference in XSTest (McNemar's p, paired prompts) and, in MT-Bench, how often the judge prefers B (one half is a tie). “Report vs list-price cost” compares the layer's reported cost for arm B with the cost independently calculated from provider-reported usage at the same versioned list prices; it is not invoice reconciliation.
Earlier weekly runs
- Sep 30, 2026 · commit ca621e2 · failed (rules broken: 4). banco-semanal-ca621e2.json
- Sep 29, 2026 · commit d650c59 · no regressions. banco-semanal-d650c59.json
- Sep 29, 2026 · commit 1100960 · no regressions. banco-semanal.json
- Sep 29, 2026 · commit 105fe13 · failed (rules broken: 7). banco-semanal-105fe13.json
Partial runs
A partial run is launched by hand with only some models or batteries; for example, the other day's share of a model whose daily provider quota does not fit its four batteries in one day. It measures what it says, but it is not the weekly run: it does not count for publishing.
- Oct 2, 2026 · commit b349d7a · google/gemini-3.1-pro-preview · GSM8K, XSTest · no regressions. banco-semanal-b349d7a-gemini-31-pro-jornada-2.json
- Oct 1, 2026 · commit b349d7a · google/gemini-3.1-pro-preview · BFCL, MT-Bench · no regressions. banco-semanal-b349d7a-gemini-31-pro-jornada-1.json
Measured savings per model
SDK A/B
GSM8K, on a request stream where 30% are exact repeats and 20% paraphrases. It is the battery with repeated traffic: your savings depend on how much yours repeats. Without repeats, see “What doesn't save”.
Both arms, model by model
GSM8K, the battery built to measure savings. Pick a measured model: A goes straight to the provider and B goes through the layer.
Measured saving per model, with its interval
SDK A/BThe dot is the estimate; the bar, the confidence interval; the line, zero. All on the same scale.
- 31.5% [25.7% – 36.7%] SDK A/B
- 200 requests · 100 unique
- 95.0% → 96.1%
- 0.4.31 · Sep 28, 2026
Only models with a measured run appear. See the methodology
| Model | n | Measured savings [CI] SDK A/B | From the exact cache | In the calls (prefix) | Accuracy A → B | Version · date |
|---|---|---|---|---|---|---|
openai/gpt-4o-mini | 200 requests · 100 unique | 31.5% [25.7% – 36.7%] | $0.0124 | $0.0006 | 95.0% → 96.1%Δ +1.1 pp [0.0, +2.8] | 0.4.31 · Sep 28, 2026 |
openai/gpt-5.4-mini | 176 requests · 88 unique | 31.7% [25.6% – 37.6%] | $0.0734 | $0.0003 | 95.6% → 93.7%Δ -1.9 pp [-5.0, 0.0] | 0.4.31 · Sep 28, 2026 |
openai/gpt-5.5 | 22 requests · 11 unique | 33.4% [6.8% – 53.8%] | $0.0790 | -$0.0023 | 100.0% → 100.0%Δ 0.0 pp [0.0, 0.0] | 0.4.31 · Sep 28, 2026 |
openai/gpt-5.6-sol | 32 requests · 16 unique | 29.2% [9.5% – 46.8%] | $0.0252 | -$0.0003 | 100.0% → 100.0%Δ 0.0 pp [0.0, 0.0] | 0.4.31 · Sep 28, 2026 |
anthropic/claude-haiku-4-5 | 116 requests · 58 unique | 32.2% [23.6% – 40.2%] | $0.0773 | $0.0002 | 99.0% → 99.0%Δ 0.0 pp [0.0, 0.0] | 0.4.31 · Sep 28, 2026 |
anthropic/claude-opus-5-5 | 32 requests · 16 unique | 77.2% [67.1% – 83.9%] | $0.0974 | $0.1391 | 100.0% → 100.0%Δ 0.0 pp [0.0, 0.0] | 0.4.31 · Sep 28, 2026 |
anthropic/claude-sonnet-5 | 62 requests · 31 unique | 75.1% [69.3% – 79.6%] | $0.1054 | $0.1313 | 100.0% → 100.0%Δ 0.0 pp [0.0, 0.0] | 0.4.31 · Sep 28, 2026 |
anthropic/claude-sonnet-5-5 | 62 requests · 31 unique | 77.5% [72.0% – 81.7%] | $0.0924 | $0.1289 | 100.0% → 100.0%Δ 0.0 pp [0.0, 0.0] | 0.4.31 · Sep 28, 2026 |
google/gemini-3.1-pro-preview | the layer failed: requests did not go through the layer | 0.4.31 · Sep 28, 2026 | ||||
google/gemini-3.8-flash | the layer failed: requests did not go through the layer | 0.4.31 · Sep 28, 2026 | ||||
Where the savings come from, position by position: “from the exact cache” is what A paid on the requests the layer served without calling the provider; “in the calls” is the difference on the ones that did reach it, mostly the provider's prefix cache (plus some output variation). Together they make the total.
Accuracy is measured on the questions with a published answer; the variants with a changed number only serve to check that the layer never hands back another question's answer.
The gateway's cache downgrade: a simulation, not an A/B
Gateway replay
This is a counterfactual replay with a partially reconstructed baseline, not one of this page's live A/B batteries. We took provider-reported usage from real Claude Code sessions at a single organization, our own, with one provider (Anthropic), and estimated costs at versioned list prices with and without PRO's cache TTL change. It is not a live double-run or invoice reconciliation.
The verdict of the Oct 3, 2026 report: the downgrade leaves a net +2.49% of the bill ($113.45 out of $4,563.61), after subtracting the cache misses it causes itself.
- Sessions
- 51
- Requests
- 19,779
- Days
- 13
| 95% interval | Net, as a share of the bill Gateway replay |
|---|---|
| Resampling by session (clusters: 51) | +1.50% – +3.57% |
| Resampling by day (clusters: 13) | +1.73% – +4.82% |
Bootstrap with 10,000 resamples. Both intervals exclude zero: on this traffic, the downgrade makes money. The card on the home page quotes the per-session interval.
The result, with what it subtracts
| Downgrades | 2,583 |
| Premium saved, gross | +3.10% |
| Cache misses caused | 124 |
| What those misses cost | -0.62% |
| Net | +2.49% · $113.45 |
| Ceiling: the whole one-hour premium | 7.2% |
Percentages are shares of the list-price bill for the simulated requests ($4,563.61). The ceiling is the most any TTL policy could recover, even if it never missed. Each percentage is rounded separately.
By model
| Model | Requests | Downgrades | Cache misses caused | Net, as a share of the model's bill |
|---|---|---|---|---|
claude-fable-5 | 1,921 | 23 | 1 | +4.64% |
claude-fable-5-1 | 2,236 | 25 | 0 | +1.65% |
claude-opus-5 | 8,344 | 70 | 2 | +1.61% |
claude-opus-4-8 | 1,193 | 34 | 0 | +6.49% |
claude-sonnet-5 | 6,080 | 2,427 | 121 | +1.34% |
claude-haiku-4-5 | 5 | 4 | 0 | no percentage (requests: 5) |
claude-sonnet-5 is the ugly case: it accounts for 94% of the downgrades and 98% of the misses, and its net is close to nothing. In the first period of the capture it actually lost: 9.4% of its bill. Its traffic is short threads that are rebuilt often and come back after the five minutes have passed. The rule really pays where session starts are expensive and rare.
How it was done: the cache model, declared
- The rule is the one the gateway runs, not a copy: it moves cache writes from the one-hour TTL to the five-minute one when the new part is at least 30% of the request body.
- The baseline is what the provider counted, with its real usage, where the downgrade was not yet active; where it already was, it is rebuilt by undoing it (see the limits). With the downgrade, the downgraded turn pays the five-minute write premium instead of the one-hour one.
- A caused miss: if the next turn of the same thread arrives after the five minutes and before the hour, what the downgraded turn wrote is no longer read from the cache, it is written again, and that cost is subtracted.
- There is a conservative variant, which also treats the prefix shared between different threads as lost. It barely moves the figure: +2.48%, with a few more misses (3).
- Aggregate figures only: no prompt, response, header or session identifier leaves the capture.
Limits
- One organization, one client, one provider. Claude Code against Anthropic, on our own traffic. No other coding agent and no other provider.
- List price, simulated. It is not an invoice: it is the real usage, priced with the price catalog. Discounts, account tiers or a subscription would change the money.
- The cache model is a model. It assumes how the provider indexes its cache entries; it does not observe it. The conservative variant bounds that risk.
- Most of the baseline is reconstructed. 79% of the requests were recorded with the downgrade already active; there the baseline is rebuilt by undoing the real downgrade, and it is less firm than that of the first stretch. That first stretch, on its own, gives a much wider interval: +0.29% to +9.12%.
- The ceiling changes with the client. It depends on how many one-hour writes the coding agent sends, and recent versions of Claude Code send fewer: of the 19,779 requests, 7,327 carried none to downgrade.
- The gap between turns is measured from the arrival of each request, not from the end of the response: it counts a few misses too many, never too few.
- It is not a forecast for your team. It is a small saving, from one traffic pattern, and other traffic can give another figure, including a worse one: look at the ugly case.
In production: since version 0.5.1 (Oct 3, 2026), the gateway subtracts the misses it observes, on the turn that suffers them, so the savings from this downgrade shown in your dashboard are net. What was recorded earlier is not recomputed, and the dashboard labels machines running an earlier gateway as “gross”.
Guarantees, counted
SDK A/B
The layer never changed a tool call
0 altered calls in 1,325 BFCL comparisons, 6 models, since 0.4.31. If the layer fails (an error in B that A did not have), it is not counted here: it is under “Bugs the bench found”.
How it is counted: every comparison of every live run since that version, so a model measured twice (a pilot and the full run) adds both. That is why the figure can exceed the n of the tables, which show one run per model.
No answer from another question since 0.4.31
0 of 198 variants (the same question with another number, hence another answer) served with another question's answer, across 8 live models and every cache configuration measured.
Offline replay of the whole GSM8K test split: 0 in 5,813 requests, in each of the 5 cache configurations.
The same refusal rate, with and without the layer
XSTest, 10 models measured: no A/B difference below p = 0.05 (McNemar, paired prompts). The layer delivered the text of 4,668 responses unaltered (0 altered).
How it is counted: responses, not prompts. Every arm-B response is compared with the provider's text, repeats of the same prompt included, so there are more responses than prompts in the table.
| Model | Prompts | Refusals, safe prompts (A → B) | Refusals, unsafe prompts (A → B) | Version · date |
|---|---|---|---|---|
openai/gpt-4o-mini | 450 | 5.2% → 5.2% (n = 250, p = 1.00) | 81.5% → 81.5% (n = 200, p = 1.00) | 0.4.31 · Sep 28, 2026 |
openai/gpt-5.4-mini | 441 | 11.0% → 9.8% (n = 245, p = 0.63) | 87.2% → 86.7% (n = 196, p = 1.00) | 0.4.31 · Sep 28, 2026 |
openai/gpt-5.5 | 120 | 6.3% → 4.7% (n = 64, p = 1.00) | 80.0% → 80.0% (n = 55, p = 1.00) | 0.4.31 · Sep 29, 2026 |
openai/gpt-5.6-sol | 199 | 2.7% → 2.7% (n = 111, p = 1.00) | 75.6% → 76.7% (n = 86, p = 1.00) | 0.4.31 · Sep 29, 2026 |
anthropic/claude-haiku-4-5 | 450 | 1.6% → 2.0% (n = 250, p = 1.00) | 42.5% → 42.5% (n = 200, p = 1.00) | 0.4.31 · Sep 29, 2026 |
anthropic/claude-opus-5-5 | 81 | — | 67.1% → 67.1% (n = 79, p = 1.00) | 0.4.31 · Sep 29, 2026 |
anthropic/claude-sonnet-5 | 264 | 2.7% → 2.0% (n = 147, p = 1.00) | 68.4% → 71.8% (n = 117, p = 0.34) | 0.4.31 · Sep 29, 2026 |
anthropic/claude-sonnet-5-5 | 311 | 1.5% → 0.0% (n = 135, p = 0.50) | 64.3% → 65.3% (n = 98, p = 1.00) | 0.4.31 · Sep 29, 2026 |
google/gemini-3.1-pro-preview | 58 | 0.0% → 0.0% (n = 32, p = 1.00) | 80.8% → 84.6% (n = 26, p = 1.00) | 0.4.31 · Sep 29, 2026 |
google/gemini-3.8-flash | 450 | 0.4% → 0.4% (n = 249, p = 1.00) | 67.8% → 66.8% (n = 199, p = 0.73) | 0.4.31 · Sep 29, 2026 |
What doesn't save
SDK A/B
Without repeats, savings are ≈ 0%. We publish it anyway, because it is what you will see if your traffic does not repeat.
MT-Bench is two-turn chat with a fresh layer per question: there is nothing for the cache to reuse, and the prompt almost never reaches the minimum the provider requires to cache a prefix. In BFCL every tool call is different. What these two batteries measure is that the layer makes nothing worse; the cost difference left over is noise in the model's output, not the layer.
| Battery | Model | n | Measured savings [CI] SDK A/B | Reading | Version · date |
|---|---|---|---|---|---|
| BFCL | openai/gpt-4o-mini | 400 items | 0.1% [0.0% – 0.3%] | indistinguishable from zero | 0.4.31 · Sep 28, 2026 |
| BFCL | openai/gpt-5.4-mini | 400 items | -0.2% [-1.0% – 0.5%] | indistinguishable from zero | 0.4.31 · Sep 28, 2026 |
| BFCL | anthropic/claude-haiku-4-5 | 48 items | 0.1% [0.0% – 0.2%] | indistinguishable from zero | 0.4.31 · Sep 29, 2026 |
| BFCL | anthropic/claude-sonnet-5 | 48 items | 1.5% [-1.6% – 5.8%] | indistinguishable from zero | 0.4.31 · Sep 29, 2026 |
| BFCL | google/gemini-3.1-pro-preview | 97 items | 4.8% [-6.6% – 15.6%] | indistinguishable from zero | 0.4.31 · Sep 29, 2026 |
| BFCL | google/gemini-3.8-flash | 324 items | -10.7% [-27.8% – 3.2%] | indistinguishable from zero | 0.4.31 · Sep 29, 2026 |
| MT-Bench | openai/gpt-4o-mini | 80 questions | 3.3% [0.8% – 6.0%] | savings · not attributable to the layer | 0.4.31 · Sep 28, 2026 |
| MT-Bench | openai/gpt-5.4-mini | 80 questions | -3.5% [-8.6% – 1.6%] | indistinguishable from zero · not attributable to the layer | 0.4.31 · Sep 28, 2026 |
| MT-Bench | openai/gpt-5.5 | 8 questions | 2.1% [-4.4% – 9.7%] | indistinguishable from zero | 0.4.31 · Sep 29, 2026 |
| MT-Bench | openai/gpt-5.6-sol | 13 questions | -0.4% [-21.2% – 13.5%] | indistinguishable from zero · not attributable to the layer | 0.4.31 · Sep 29, 2026 |
| MT-Bench | anthropic/claude-haiku-4-5 | 80 questions | -2.6% [-4.6% – -0.9%] | extra cost · not attributable to the layer | 0.4.31 · Sep 28, 2026 |
| MT-Bench | anthropic/claude-opus-5-5 | 18 questions | 1.1% [-2.0% – 4.9%] | indistinguishable from zero | 0.4.31 · Sep 29, 2026 |
| MT-Bench | anthropic/claude-sonnet-5 | 23 questions | 1.7% [-8.9% – 12.7%] | indistinguishable from zero | 0.4.31 · Sep 28, 2026 |
| MT-Bench | anthropic/claude-sonnet-5-5 | 37 questions | -0.8% [-2.5% – 0.9%] | indistinguishable from zero | 0.4.31 · Sep 29, 2026 |
Status per model and battery
Each cell shows the run on the layer's highest version. If a model could not be measured, we say why instead of leaving a gap.
| Model | GSM8K | BFCL | XSTest | MT-Bench |
|---|---|---|---|---|
openai/gpt-4o-mini | measured · n = 200 0.4.31 · Sep 28, 2026 | measured · n = 400 0.4.31 · Sep 28, 2026 | measured · n = 450 0.4.31 · Sep 28, 2026 | measured · n = 80 0.4.31 · Sep 28, 2026 |
openai/gpt-5.4-mini | measured · n = 176 0.4.31 · Sep 28, 2026 | measured · n = 400 0.4.31 · Sep 28, 2026 | measured · n = 441 0.4.31 · Sep 28, 2026 | measured · n = 80 0.4.31 · Sep 28, 2026 |
openai/gpt-5.5 | measured · n = 22 0.4.31 · Sep 28, 2026 | not measured: spend cap exhausted 0.4.31 · Sep 29, 2026 | measured · n = 120 0.4.31 · Sep 29, 2026 | measured · n = 8 0.4.31 · Sep 29, 2026 |
openai/gpt-5.6-sol | measured · n = 32 0.4.31 · Sep 28, 2026 | not measured: the provider rejected the request in both arms 0.4.31 · Sep 29, 2026 | measured · n = 199 0.4.31 · Sep 29, 2026 | measured · n = 13 0.4.31 · Sep 29, 2026 |
anthropic/claude-haiku-4-5 | measured · n = 116 0.4.31 · Sep 28, 2026 | measured · n = 48 0.4.31 · Sep 29, 2026 | measured · n = 450 0.4.31 · Sep 29, 2026 | measured · n = 80 0.4.31 · Sep 28, 2026 |
anthropic/claude-opus-5-5 | measured · n = 32 0.4.31 · Sep 28, 2026 | not measured: spend cap exhausted 0.4.31 · Sep 29, 2026 | measured · n = 81 0.4.31 · Sep 29, 2026 | measured · n = 18 0.4.31 · Sep 29, 2026 |
anthropic/claude-sonnet-5 | measured · n = 62 0.4.31 · Sep 28, 2026 | measured · n = 48 0.4.31 · Sep 29, 2026 | measured · n = 264 0.4.31 · Sep 29, 2026 | measured · n = 23 0.4.31 · Sep 28, 2026 |
anthropic/claude-sonnet-5-5 | measured · n = 62 0.4.31 · Sep 28, 2026 | not measured: spend cap exhausted 0.4.31 · Sep 29, 2026 | measured · n = 311 0.4.31 · Sep 29, 2026 | measured · n = 37 0.4.31 · Sep 29, 2026 |
google/gemini-3.1-pro-preview | the layer failed: requests did not go through the layer 0.4.31 · Sep 28, 2026 | measured · n = 97 0.4.31 · Sep 29, 2026 | measured · n = 58 0.4.31 · Sep 29, 2026 | not measured: provider quota exhausted 0.4.31 · Sep 29, 2026 |
google/gemini-3.8-flash | the layer failed: requests did not go through the layer 0.4.31 · Sep 28, 2026 | measured · n = 324 0.4.31 · Sep 29, 2026 | measured · n = 450 0.4.31 · Sep 29, 2026 | the layer failed: requests did not go through the layer 0.4.31 · Sep 29, 2026 |
Compatibility
Compatible is not measured. The SDK works with any endpoint compatible with the OpenAI API, plus the native Anthropic adapter and the native Gemini adapter (new in 0.4.33). With a savings figure on the public bench, today only OpenAI and Anthropic models; Gemini has no savings figure. What is measured for Gemini: Gemini 3.8 Flash, through its native API, in the weekly live bench of October 1, 2026 — all four batteries, with no layer failures and no refusal cached or credited as savings. That bench checks that the layer breaks nothing and does not overcharge; it does not measure a saving. Also measured: Gemini 3.1 Pro, through its native API, in two partial runs on October 1 – 2, 2026 (its daily provider quota does not fit all four batteries in one day): all four batteries, again with no layer failures and no refusal cached or credited as savings.
Bugs the bench found
The bench is not there to look good: it is there to find the bugs before you do. It found these, with the version they appeared in and the one that fixes them.
The uncalibrated semantic cache served another question's answer 0.4.30 → 0.4.31
With the semantic cache uncalibrated, a variant with another number (“three baskets” versus “two baskets”) could receive the original question's answer. Counted over the GSM8K variants.
- Found in 0.4.30: 15 of 111. gsm8k.json
- Fixed in 0.4.31 (#383).
- Verified live on 0.4.31: 0 of 198.
The layer trimmed the tool catalog with no signal to do so 0.4.30 → 0.4.31
In multi-turn conversations, arm B sent fewer tools than arm A even though nothing indicated which ones were unneeded. Counted over the BFCL multi-turn conversations.
- Found in 0.4.30: 20 of 20. bfcl.json
- Fixed in 0.4.31 (#382).
- Verified live on 0.4.31: 0 of 83.
Some refusals written as text were cached 0.4.31 → 0.4.32
A refusal the model writes as text (without the API's signal) could be cached, repeated and credited as savings. Counted in XSTest prompts.
- Found in 0.4.31: 39 of 2,824. xstest-0431-claude.jsonxstest-0431-gama-alta.jsonxstest-0431.json
- Fixed in 0.4.32 (#393, #401).
- Re-measured live by the weekly bench on Oct 1, 2026 (source of main with the fix, commit 5bec174): 0 refusals cached and 0 credited, over 2,393 XSTest prompts on 9 models. banco-semanal-5bec174.json
Partial refusals were cached and credited 0.4.32 → 0.4.33
A response that starts by complying and refuses further on (“If you mean harming someone, I can't help…”), or an apology followed by “I don't have access…”, was not recognized as a refusal: the layer could cache it, repeat it and credit it as savings. Counted in the weekly bench's XSTest prompts, with OpenAI and Claude models.
- Found by the weekly live bench on Sep 29, 2026 (source of main, commit 105fe13): 12 refusals served from or written to the cache and 14 credited, over 1,995 XSTest prompts on 8 models. banco-semanal-105fe13.json
- Fixed in 0.4.33 (#422).
- Re-measured live by the weekly bench on Oct 1, 2026 (source of main with the fix, commit 5bec174): 0 refusals cached and 0 credited, over 2,393 XSTest prompts on 9 models. banco-semanal-5bec174.json
A Gemini content filter was cached as an answer 0.4.31 → 0.4.32
Gemini flags its filter with its own stop reason, which the layer did not recognize as a refusal, so it could cache and repeat it. Counted in XSTest prompts.
- Found in 0.4.31: 6 of 508. xstest-0431-gama-alta.json
- Fixed in 0.4.32 (#402).
- Re-measured live by the weekly bench on Oct 1, 2026 (source of main with the fix, commit 5bec174) with google/gemini-3.8-flash: 0 refusals cached or credited, over 448 XSTest prompts. That run could not measure google/gemini-3.1-pro-preview (the provider's quota or rate limit ran out, not the layer); there it stays unverified. banco-semanal-5bec174.json
Gemini did not go through the layer 0.4.31 → 0.4.32
The layer sent a field specific to the OpenAI API to every compatible endpoint, and Gemini rejects it: every request in arm B failed. Counted in GSM8K requests.
- Found in 0.4.31: 504 of 504. gsm8k-0431-gama-alta.json
- Fixed in 0.4.32 (#402).
- Re-measured live by the weekly bench on Oct 1, 2026 (source of main with the fix, commit 5bec174) with google/gemini-3.8-flash: 0 errors only with the layer, over 68 GSM8K items. That run could not measure google/gemini-3.1-pro-preview (the provider's quota or rate limit ran out, not the layer); there it stays unverified. banco-semanal-5bec174.json
With Gemini 3, the second turn with tools failed 0.4.31 → 0.4.32
The layer did not return the thought signature Gemini 3 requires on each tool call, so the next turn was rejected. Counted in BFCL multi-turn continuations.
- Found in 0.4.31: 50 of 50. bfcl-0431-gama-alta.json
- Fixed in 0.4.32 (#402).
- Not re-measured in a published report yet: the weekly bench of Oct 1, 2026 (source of main with the fix, commit 5bec174) measured google/gemini-3.8-flash in this battery, but its summary counts only single-turn entries, not the multi-turn continuations where this failed. Until a report counts them, we do not count it as verified. That run could not measure google/gemini-3.1-pro-preview (the provider's quota or rate limit ran out, not the layer); there it stays unverified. banco-semanal-5bec174.json
With requests in parallel, Gemini's thinking-level correction reached the client as an error 0.4.35 → 0.4.36
Gemini Flash rejects the MINIMAL thinking level; the native adapter retries once with LOW and remembers the substitute. With two requests in flight through the same layer, one compared the rejection with the body the other had already corrected, did not recognize it and passed the error to the client. Counted as errors only with the layer, in the weekly bench's XSTest prompts with Gemini.
- Found by the weekly live bench on Sep 30, 2026 (source of main, commit ca621e2) with google/gemini-3.8-flash: 1 errors only with the layer, over 450 XSTest items. banco-semanal-ca621e2.json
- Fixed in 0.4.36 (#442).
- Re-measured live by the weekly bench on Oct 1, 2026 (source of main with the fix, commit 5bec174) with google/gemini-3.8-flash: 0 errors only with the layer, over 448 XSTest items. That run could not measure google/gemini-3.1-pro-preview (the provider's quota or rate limit ran out, not the layer); there it stays unverified. banco-semanal-5bec174.json
How to reproduce it
The bench's code is in the open repository, and the method fits in five steps. With your own provider keys you can rerun it and compare figure by figure with our reports.
- The dataset. Download each public dataset at the commit in the table above and check every file's sha256 before using it. If it does not match, it is not the same dataset.
- The layer. Install
@bivelio/savings-layerfrom npm at the version the report names and check the sha256 ofdist/index.js: every report carries its own. - Two arms per prompt. A goes straight to the provider; B sends the same prompt, to the same model and with the same output cap, through the layer with its default configuration. Part of the prompts are repeated or paraphrased, as in real traffic, and the A/B order is randomised.
- Usage and list prices document the valuation. Add each arm's cost from provider-reported
usageat the versioned list price. Savings are1 − cost B / cost A, with their interval; this is not invoice reconciliation. - Quality, in the same run. Score both arms' answers the same way (exact answer in GSM8K, tool call in BFCL, refusal or not in XSTest, a judge in MT-Bench) and compare them with the JSON report for the same battery and version.
The reports
Every run leaves a JSON report with its design, its limitations and every measured call: usage, cost per arm and how the answer was classified. They are served from this website.
The reports are served as the bench wrote them, with one exception: data about the environment of whoever ran it (disk paths, email addresses) is removed. For XSTest no answer text is published, only its classification.
Measure us yourself
An independent measurement is worth more than ours. The bench's code is in the open repository: if you want to rerun the measurement, or measure the layer with your own harness or your own traffic, write to us and we will give you what you need to do it.
Write to us at support@bivelio.com