Our A/B bench, in the open. Measure us.

What is measured, and how

• Two arms, the same request. A goes straight to the provider, without the layer; B goes through the layer with its default configuration. Same prompt, same sampling, same output cap.
• The provider counts the cost. Each arm is priced from the usage the API returns on every call, at list price. Savings are 1 − cost(B) / cost(A) of the same run, not what the layer says about itself.
• 95% intervals. Each harness computes them (resampling whole question families, Wilson for refusal rates). An interval that crosses zero means “indistinguishable from zero”, and we say so.
• The package you install. Arm B runs the source of tag 0.4.31; its dist/index.js is byte-identical to the npm tarball's (sha256 b52f99cacf9804ecd3153e56f26ec3adcdc2510d52a8446a369ecf914e0b5731).
• No shortcuts. Each harness watches the hosts it talks to and carries a per-model spend cap written in code. A model that could not be measured is reported as “not measured”, with the reason.

Four batteries, public datasets, pinned

Measured savings per model

GSM8K, on a request stream where 30% are exact repeats and 20% paraphrases. It is the battery with repeated traffic: your savings depend on how much yours repeats. Without repeats, see “What doesn't save”.

Where the savings come from, position by position: “from the exact cache” is what A paid on the requests the layer served without calling the provider; “in the calls” is the difference on the ones that did reach it, mostly the provider's prefix cache (plus some output variation). Together they make the total.

Accuracy is measured on the questions with a published answer; the variants with a changed number only serve to check that the layer never hands back another question's answer.

Guarantees, counted

What doesn't save

Without repeats, savings are ≈ 0%. We publish it anyway, because it is what you will see if your traffic does not repeat.

MT-Bench is two-turn chat with a fresh layer per question: there is nothing for the cache to reuse, and the prompt almost never reaches the minimum the provider requires to cache a prefix. In BFCL every tool call is different. What these two batteries measure is that the layer makes nothing worse; the cost difference left over is noise in the model's output, not the layer.

Status per model and battery

Each cell shows the run on the layer's highest version. If a model could not be measured, we say why instead of leaving a gap.

Bugs the bench found

The bench is not there to look good: it is there to find the bugs before you do. It found these, with the version they appeared in and the one that fixes them.

1. The uncalibrated semantic cache served another question's answer 0.4.30 → 0.4.31

   With the semantic cache uncalibrated, a variant with another number (“three baskets” versus “two baskets”) could receive the original question's answer. Counted over the GSM8K variants.

   • Found in 0.4.30: 15 of 111. gsm8k.json
   • Fixed in 0.4.31 (#383).
   • Verified live on 0.4.31: 0 of 198.

2. The layer trimmed the tool catalog with no signal to do so 0.4.30 → 0.4.31

   In multi-turn conversations, arm B sent fewer tools than arm A even though nothing indicated which ones were unneeded. Counted over the BFCL multi-turn conversations.

   • Found in 0.4.30: 20 of 20. bfcl.json
   • Fixed in 0.4.31 (#382).
   • Verified live on 0.4.31: 0 of 83.

3. Some refusals written as text were cached 0.4.31 → 0.4.32

   A refusal the model writes as text (without the API's signal) could be cached, repeated and credited as savings. Counted in XSTest prompts.

   • Found in 0.4.31: 39 of 2,824. xstest-0431-claude.jsonxstest-0431-gama-alta.jsonxstest-0431.json
   • Fixed in 0.4.32 (#393, #401).
   • Waiting to be re-measured live on 0.4.32.

4. A Gemini content filter was cached as an answer 0.4.31 → 0.4.32

   Gemini flags its filter with its own stop reason, which the layer did not recognize as a refusal, so it could cache and repeat it. Counted in XSTest prompts.

   • Found in 0.4.31: 6 of 508. xstest-0431-gama-alta.json
   • Fixed in 0.4.32 (#402).
   • Waiting to be re-measured live on 0.4.32.

5. Gemini did not go through the layer 0.4.31 → 0.4.32

   The layer sent a field specific to the OpenAI API to every compatible endpoint, and Gemini rejects it: every request in arm B failed. Counted in GSM8K requests.

   • Found in 0.4.31: 504 of 504. gsm8k-0431-gama-alta.json
   • Fixed in 0.4.32 (#402).
   • Waiting to be re-measured live on 0.4.32.

6. With Gemini 3, the second turn with tools failed 0.4.31 → 0.4.32

   The layer did not return the thought signature Gemini 3 requires on each tool call, so the next turn was rejected. Counted in BFCL multi-turn continuations.

   • Found in 0.4.31: 50 of 50. bfcl-0431-gama-alta.json
   • Fixed in 0.4.32 (#402).
   • Waiting to be re-measured live on 0.4.32.

How to reproduce it

The bench's code is not public yet; the method is, and it fits in five steps. With your own provider keys you can rerun it and compare figure by figure with our reports.

1. The dataset. Download each public dataset at the commit in the table above and check every file's sha256 before using it. If it does not match, it is not the same dataset.
2. The layer. Install @bivelio/savings-layer from npm at the version the report names and check the sha256 of dist/index.js: every report carries its own.
3. Two arms per prompt. A goes straight to the provider; B sends the same prompt, to the same model and with the same output cap, through the layer with its default configuration. Part of the prompts are repeated or paraphrased, as in real traffic, and the A/B order is randomised.
4. The provider states the cost. Add up what each arm is billed according to the usage the provider returns, at its published price. Savings are 1 − cost B / cost A, with their interval.
5. Quality, in the same run. Score both arms' answers the same way (exact answer in GSM8K, tool call in BFCL, refusal or not in XSTest, a judge in MT-Bench) and compare them with the JSON report for the same battery and version.

The reports

Every run leaves a JSON report with its design, its limitations and every measured call: usage, cost per arm and how the answer was classified. They are served from this website.

The reports are served as the bench wrote them, with one exception: data about the environment of whoever ran it (disk paths, email addresses) is removed. For XSTest no answer text is published, only its classification.