{"bateria":"mt-bench","fecha":"2026-09-28","modo":"LIVE","commitDelRepo":"1eb205c8b46ba4b435a56aa88c1f0576a5978423","paquete":{"nombre":"@bivelio/savings-layer","version":"0.4.31","integrity":"sha512-ePCN+dZr+VuMUSGm2TelDkt7TR6jSi/SCTomomDK/bfF92FHr1eFUdgTWQODCdQf8C8c+/h6AA/up46DXO2mLQ== (tarball de npm; dist/index.js sha256 b52f99cacf9804ecd3153e56f26ec3adcdc2510d52a8446a369ecf914e0b5731 = build del fuente)","origen":"fuente de v0.4.31 (00c99c803c0ab6f3c11b7ab87f3c75a38d609b1c), verificado idéntico al tarball de npm"},"brazoB":"v0.4.31 (source of the tag, BVSALA_DEV_OPEN=1; npm = same commit, verified identical)","verificacionDelFuente":{"etiqueta":"v0.4.31","commit":"00c99c803c0ab6f3c11b7ab87f3c75a38d609b1c","fetchDeEtiquetas":"ok","diffDelRepo":["M src/version.ts"],"arbolCargado":"benchmarks/out/mt-bench/.fuente-v0.4.31 (git archive v0.4.31)","ficherosVerificados":173,"diffDelArbolCargado":[],"npm":{"tarball":"@bivelio/savings-layer@0.4.31 (npm pack)","integrity":"sha512-ePCN+dZr+VuMUSGm2TelDkt7TR6jSi/SCTomomDK/bfF92FHr1eFUdgTWQODCdQf8C8c+/h6AA/up46DXO2mLQ==","sha256DistIndex":"b52f99cacf9804ecd3153e56f26ec3adcdc2510d52a8446a369ecf914e0b5731"},"build":{"sha256DistIndex":"b52f99cacf9804ecd3153e56f26ec3adcdc2510d52a8446a369ecf914e0b5731"},"distIdentico":true,"versionCargada":"0.4.31"},"dataset":{"origen":"fastchat-fijado","repo":"lm-sys/FastChat","commit":"587d5cfa1609a43d192cedb8441cac3c17db105d","license":"Apache-2.0","ficheros":{"preguntas":{"path":"fastchat/llm_judge/data/mt_bench/question.jsonl","sha256":"119565adbab82227089cefdb44c8d7e2cf04dc0a0ec233634c82e7d4e2a944f7"},"referencias":{"path":"fastchat/llm_judge/data/mt_bench/reference_answer/gpt-4.jsonl","sha256":"f957a5bc977badb66885ec970e6cd08527845780313f0995764260e5777b9b3f"},"promptsDelJuez":{"path":"fastchat/llm_judge/data/judge_prompts.jsonl","sha256":"fd283293406d024f44c174b094ef48031d0687a4682fd3a56b29b138f80281b6"}}},"juez":{"modelo":"openai/gpt-4o","topeUsd":0.5},"topes":{"porFamiliaUsd":{"openai":0.8,"anthropic":1.25},"juezTotalUsd":0.5},"semilla":20260928,"gastoTotal":{"generacion":1.8913510000000007,"juez":0.40020500000000003},"gastoAcumuladoEntreTiradas":{"generacion":{"claude-haiku-4-5":0.7688210000000003,"claude-sonnet-5":1.1225300000000005},"juez":{"claude-haiku-4-5":0.15779,"claude-sonnet-5":0.242415}},"hosts":{"permitidos":["api.anthropic.com","api.openai.com","ai-gateway.vercel.sh","raw.githubusercontent.com"],"vistos":{"raw.githubusercontent.com":3,"api.anthropic.com":413,"ai-gateway.vercel.sh":54},"bloqueados":{}},"modelos":[{"modelo":"claude-haiku-4-5","idCualificado":"anthropic/claude-haiku-4-5","familia":"anthropic","maxSalida":1024,"temperatura":"FastChat por categoría (igual en A y B)","precio":{"inputPer1k":0.001,"outputPer1k":0.005,"cachedInputPer1k":0.0001},"precioConocido":true,"preguntas":{"planificadas":80,"completas":80,"conError":0,"porCategoria":{"writing":10,"roleplay":10,"reasoning":10,"math":10,"coding":10,"extraction":10,"stem":10,"humanities":10},"cortePorPresupuesto":false},"gasto":{"generacionA":0.3794000000000001,"generacionB":0.38942099999999996,"total":0.7688210000000003,"tope":1.25,"excesosDeReserva":0,"juez":0.15779,"topeJuez":0.15779},"coste":{"A":0.3794,"B":0.389421,"ahorroPooled":{"point":-0.026412756984712793,"lower":-0.04628024673153086,"upper":-0.00928840617571658},"ahorroEstratificadoPorCategoria":{"point":-0.02641275698471257,"lower":-0.044120116635879336,"upper":-0.010403935283226362},"clase":"sobrecoste","atribucion":{"atribuible":false,"motivo":"el cable de B solo difiere de A en `messages:cache_control`, las marcas `cache_control` de B no escribieron ni leyeron un token (tramo por debajo del mínimo del modelo: el proveedor las ignora), B no leyó de caché más que A (0 frente a 0 llamadas) y la capa declara 0 $ de ahorro: la diferencia de coste es muestreo de la salida, no la capa"},"controlAA":{"preguntas":80,"ahorro":{"point":-0.029655879301509147,"lower":-0.058573479420044317,"upper":-0.005818103940605113},"clase":"sobrecoste"},"deltaMedioPorPreguntaUsd":{"point":-0.00012526250000000012,"lower":-0.00021690000000000007,"upper":-0.00004362500000000002},"wilcoxonP":0.9896357109387275},"tokens":{"A":{"entrada":45605,"cacheLeida":0,"cacheEscrita":0,"salida":66759,"razonamiento":0},"B":{"entrada":46556,"cacheLeida":0,"cacheEscrita":0,"salida":68573,"razonamiento":0}},"cacheDePrefijo":{"llamadas":320,"llamadasConLecturaA":0,"llamadasConLecturaB":0,"llamadasConEscrituraA":0,"llamadasConEscrituraB":0,"llamadasConMarcaA":0,"llamadasConMarcaB":80,"primaDeEscrituraUsdA":0,"primaDeEscrituraUsdB":0,"ahorroPorLecturasUsdA":0,"ahorroPorLecturasUsdB":0},"latencia":{"A":{"p50":3915,"p95":10623},"B":{"p50":4192,"p95":10897},"deltaMedioMs":{"point":78.075,"lower":-21.01875,"upper":174.25},"llamadasSinReintento":320},"truncadas":{"A":27,"B":29},"cable":{"turnos":160,"identicos":80,"diferencias":{"messages:cache_control":80},"modelosServidosA":["claude-haiku-4-5-20251001"],"modelosServidosB":["claude-haiku-4-5-20251001"]},"capa":{"basis":{"measured":160},"ahorroDeclaradoUsd":0,"turnosConAhorroDeclaradoDistintoDe0":0,"ahorroDeclaradoPorBasis":{"measured":0},"costeDeclaradoUsd":0.38942099999999996,"costeMedidoB":0.389421,"tokensQueNoCuadran":0,"cacheResult":{"miss":160},"executionClass":{"NO_RETRIEVAL":50,"SINGLE_RETRIEVAL":83,"MULTI_HOP_RETRIEVAL":27},"llamadasExtraAlProveedor":0,"etapas":{"canonicalize:done":160,"policy:stochastic sampling (temp>0.3, no seed)":12,"router:NO_RETRIEVAL":50,"constraints:none":160,"reference:written":130,"policy:freshness-sensitive request; personalized/account-scoped request; stochastic sampling (temp>0.3, no seed); freshness-sensitive without a freshness bucket → exact cache off; personalized request with no principal (context.sessionId) → exact cache off; agentic request (tools, tool calls/results or more than one user turn) → semantic cache off by default (opt in with savings.semanticCache: \"include-agentic\")":6,"router:SINGLE_RETRIEVAL":83,"prefix-cache:marked":80,"prefix-cache:measured":80,"policy:personalized/account-scoped request; stochastic sampling (temp>0.3, no seed); personalized request with no principal (context.sessionId) → exact cache off":6,"reference:not-written":30,"cache-write:skipped: the answer reads as a refusal written as text (text_refusal); it was measured and served as usual, but a refusal is not an answer to replay or to credit as an avoided call":1,"policy:freshness-sensitive request; freshness-sensitive without a freshness bucket → exact cache off":3,"policy:freshness-sensitive request; freshness-sensitive without a freshness bucket → exact cache off; agentic request (tools, tool calls/results or more than one user turn) → semantic cache off by default (opt in with savings.semanticCache: \"include-agentic\")":20,"policy:no restrictions":52,"exact-cache:miss":81,"elision:skipped":81,"policy:agentic request (tools, tool calls/results or more than one user turn) → semantic cache off by default (opt in with savings.semanticCache: \"include-agentic\")":29,"cache-write:skipped: answer cut off by the output cap (finish_reason \"length\"); replaying half an answer would bill it as a 100% avoided call":29,"router:MULTI_HOP_RETRIEVAL":27,"policy:personalized/account-scoped request; personalized request with no principal (context.sessionId) → exact cache off":5,"policy:personalized/account-scoped request; personalized request with no principal (context.sessionId) → exact cache off; agentic request (tools, tool calls/results or more than one user turn) → semantic cache off by default (opt in with savings.semanticCache: \"include-agentic\")":8,"policy:personalized/account-scoped request; stochastic sampling (temp>0.3, no seed); personalized request with no principal (context.sessionId) → exact cache off; agentic request (tools, tool calls/results or more than one user turn) → semantic cache off by default (opt in with savings.semanticCache: \"include-agentic\")":6,"policy:freshness-sensitive request; personalized/account-scoped request; freshness-sensitive without a freshness bucket → exact cache off; personalized request with no principal (context.sessionId) → exact cache off; agentic request (tools, tool calls/results or more than one user turn) → semantic cache off by default (opt in with savings.semanticCache: \"include-agentic\")":3,"policy:stochastic sampling (temp>0.3, no seed); agentic request (tools, tool calls/results or more than one user turn) → semantic cache off by default (opt in with savings.semanticCache: \"include-agentic\")":4,"policy:freshness-sensitive request; stochastic sampling (temp>0.3, no seed); freshness-sensitive without a freshness bucket → exact cache off":2,"policy:freshness-sensitive request; stochastic sampling (temp>0.3, no seed); freshness-sensitive without a freshness bucket → exact cache off; agentic request (tools, tool calls/results or more than one user turn) → semantic cache off by default (opt in with savings.semanticCache: \"include-agentic\")":4}},"calidad":{"juzgada":true,"turnosJuzgados":15,"victoriasB":0,"victoriasA":0,"empates":15,"inconsistentes":6,"identicas":4,"errores":0,"turnosSinPresupuesto":1,"tasaDeVictoriaB":{"point":0.5,"lower":0.5,"upper":0.5},"porCategoria":{"writing":{"n":2,"B":0,"A":0,"empate":2},"roleplay":{"n":2,"B":0,"A":0,"empate":2},"reasoning":{"n":2,"B":0,"A":0,"empate":2},"math":{"n":2,"B":0,"A":0,"empate":2},"coding":{"n":2,"B":0,"A":0,"empate":2},"extraction":{"n":2,"B":0,"A":0,"empate":2},"stem":{"n":2,"B":0,"A":0,"empate":2},"humanities":{"n":1,"B":0,"A":0,"empate":1}},"llamadasAlJuez":23,"preguntasJuzgadas":8,"cortePorPresupuesto":true},"respuestasIdenticasPorTurno":30,"errores":[]},{"modelo":"claude-sonnet-5","idCualificado":"anthropic/claude-sonnet-5","familia":"anthropic","maxSalida":4096,"temperatura":"no se envía (modelo de razonamiento), en ningún brazo","precio":{"inputPer1k":0.002,"outputPer1k":0.01,"cachedInputPer1k":0.0002},"precioConocido":true,"preguntas":{"planificadas":80,"completas":23,"conError":0,"porCategoria":{"writing":3,"roleplay":3,"reasoning":3,"math":3,"coding":3,"extraction":3,"stem":3,"humanities":2},"cortePorPresupuesto":true,"motivoDelCorte":"el peor coste de la pregunta siguiente ($0.2318) no cabía: gastado $1.0281 de $1.2500"},"gasto":{"generacionA":0.5659359999999999,"generacionB":0.556594,"total":1.1225300000000005,"tope":1.25,"excesosDeReserva":0,"juez":0.242415,"topeJuez":0.242415},"coste":{"A":0.5659360000000001,"B":0.556594,"ahorroPooled":{"point":0.01650716688812881,"lower":-0.08914978653828731,"upper":0.12714219681588645},"ahorroEstratificadoPorCategoria":{"point":0.025194055596099107,"lower":-0.06397886168435107,"upper":0.10666251331483512},"clase":"indistinguible de 0","atribucion":{"atribuible":true,"motivo":"posible efecto de la capa: el cable difiere en `messages:cache_control`; la capa declara sobrecoste (-0.005032 $)"},"controlAA":{"preguntas":23,"ahorro":{"point":-0.034150517266501534,"lower":-0.15064254282157807,"upper":0.08686200887483575},"clase":"indistinguible de 0"},"deltaMedioPorPreguntaUsd":{"point":0.0004061739130434796,"lower":-0.002114195652173912,"upper":0.0031968695652173943},"wilcoxonP":0.6515792601923025},"tokens":{"A":{"entrada":24133,"cacheLeida":0,"cacheEscrita":0,"salida":51767,"razonamiento":0},"B":{"entrada":13437,"cacheLeida":0,"cacheEscrita":10064,"salida":50456,"razonamiento":0}},"cacheDePrefijo":{"llamadas":92,"llamadasConLecturaA":0,"llamadasConLecturaB":0,"llamadasConEscrituraA":0,"llamadasConEscrituraB":6,"llamadasConMarcaA":0,"llamadasConMarcaB":23,"primaDeEscrituraUsdA":0,"primaDeEscrituraUsdB":0.005032,"ahorroPorLecturasUsdA":0,"ahorroPorLecturasUsdB":0},"latencia":{"A":{"p50":10621,"p95":29615},"B":{"p50":9303,"p95":24404},"deltaMedioMs":{"point":-562.3695652173913,"lower":-1776.7608695652175,"upper":487.2391304347826},"llamadasSinReintento":92},"truncadas":{"A":1,"B":0},"cable":{"turnos":46,"identicos":23,"diferencias":{"messages:cache_control":23},"modelosServidosA":["claude-sonnet-5"],"modelosServidosB":["claude-sonnet-5"]},"capa":{"basis":{"measured":46},"ahorroDeclaradoUsd":-0.005032000000000009,"turnosConAhorroDeclaradoDistintoDe0":6,"ahorroDeclaradoPorBasis":{"measured":-0.005032000000000009},"costeDeclaradoUsd":0.556594,"costeMedidoB":0.556594,"tokensQueNoCuadran":0,"cacheResult":{"miss":46},"executionClass":{"NO_RETRIEVAL":18,"SINGLE_RETRIEVAL":21,"MULTI_HOP_RETRIEVAL":7},"llamadasExtraAlProveedor":0,"etapas":{"canonicalize:done":46,"policy:no restrictions":15,"exact-cache:miss":21,"router:NO_RETRIEVAL":18,"elision:skipped":21,"constraints:none":46,"reference:written":46,"policy:personalized/account-scoped request; personalized request with no principal (context.sessionId) → exact cache off; agentic request (tools, tool calls/results or more than one user turn) → semantic cache off by default (opt in with savings.semanticCache: \"include-agentic\")":5,"router:SINGLE_RETRIEVAL":21,"prefix-cache:marked":23,"prefix-cache:measured":23,"policy:personalized/account-scoped request; personalized request with no principal (context.sessionId) → exact cache off":6,"policy:freshness-sensitive request; personalized/account-scoped request; freshness-sensitive without a freshness bucket → exact cache off; personalized request with no principal (context.sessionId) → exact cache off; agentic request (tools, tool calls/results or more than one user turn) → semantic cache off by default (opt in with savings.semanticCache: \"include-agentic\")":7,"router:MULTI_HOP_RETRIEVAL":7,"policy:freshness-sensitive request; freshness-sensitive without a freshness bucket → exact cache off":2,"policy:freshness-sensitive request; freshness-sensitive without a freshness bucket → exact cache off; agentic request (tools, tool calls/results or more than one user turn) → semantic cache off by default (opt in with savings.semanticCache: \"include-agentic\")":5,"policy:agentic request (tools, tool calls/results or more than one user turn) → semantic cache off by default (opt in with savings.semanticCache: \"include-agentic\")":6}},"calidad":{"juzgada":true,"turnosJuzgados":17,"victoriasB":3,"victoriasA":4,"empates":10,"inconsistentes":3,"identicas":2,"errores":0,"turnosSinPresupuesto":1,"tasaDeVictoriaB":{"point":0.4722222222222222,"lower":0.3055555555555556,"upper":0.6666666666666666},"porCategoria":{"writing":{"n":3,"B":0,"A":1,"empate":2},"roleplay":{"n":2,"B":1,"A":0,"empate":1},"reasoning":{"n":2,"B":0,"A":1,"empate":1},"math":{"n":2,"B":0,"A":0,"empate":2},"coding":{"n":2,"B":0,"A":0,"empate":2},"extraction":{"n":2,"B":0,"A":0,"empate":2},"stem":{"n":2,"B":0,"A":2,"empate":0},"humanities":{"n":2,"B":2,"A":0,"empate":0}},"llamadasAlJuez":31,"preguntasJuzgadas":9,"cortePorPresupuesto":true},"respuestasIdenticasPorTurno":3,"errores":[]}],"modelosFallidos":[],"limitaciones":["MT-Bench es conversación de 2 turnos, prompts cortos (cientos de tokens) y sin herramientas ni contrato de salida: casi ninguna palanca de la capa tiene forma sobre la que actuar, y la caché de prefijo del proveedor no llega al mínimo de tramo (1024 tokens en OpenAI y en claude-sonnet-5, 4096 en claude-haiku-4-5) en la mayoría de turnos. Este banco mide sobre todo que la capa NO empeore (coste, latencia, calidad) en tráfico de chat, no cuánto ahorra en tráfico agéntico.","Una capa NUEVA por pregunta (caché y memoria frías): no hay repeticiones entre preguntas, así que la caché exacta/semántica de la capa no tiene ocasión de acertar. Es a propósito: el tráfico repetido va en otra batería.","Subconjunto estratificado con semilla cuando las 80 preguntas no caben en el tope por modelo ($0.8000 OpenAI / $1.2500 Anthropic); la calidad, además, solo se juzga en las preguntas que caben en $0.5000 de juez EN TOTAL (repartidos entre los modelos medidos), así que su intervalo es ancho.","El juez es gpt-4o por Vercel AI Gateway: sesgos conocidos de juez LLM (verbosidad, autopreferencia de familia OpenAI). Las posiciones intercambiadas controlan el sesgo de posición, no los demás.","Temperatura de FastChat por categoría (0,7 en writing/roleplay): parte de las diferencias de calidad entre A y B en esas categorías es muestreo, no la capa. Los dos brazos muestrean igual; gpt-5.4-mini y claude-sonnet-5 no aceptan temperature y no se manda en ningún brazo.","El dinero es modelado: tokens medidos × precio de lista del catálogo del paquete. Descuentos negociados, batch o niveles no entran.","Tope de salida NATIVO de la 0.4.31 en el brazo B (`maxOutputTokens` → `max_completion_tokens` en OpenAI, `max_tokens` en Anthropic) y el mismo campo y valor en el brazo A. gpt-5.4-mini y claude-sonnet-5 llevan 4096 (su razonamiento —en Sonnet 5, el pensamiento adaptativo que corre por defecto sin `thinking`— cuenta dentro); FastChat usa 1024.","Anthropic: una capa nueva por pregunta y solo 2 turnos. La capa marca `cache_control` en el turno 2 (fin del prefijo estable = la respuesta del turno 1), pero no hay turno 3 que lea esa escritura: en este banco la marca solo puede costar (prima 1,25× sobre el tramo escrito) y nunca amortizarse. En una conversación larga, esa misma marca es la que haría que el turno siguiente leyera a 0,1×; este banco no mide ese caso.","claude-sonnet-5 piensa (adaptativo) por defecto y el pensamiento se omite en la respuesta (`display: omitted`) pero se factura dentro de `output_tokens`; el juez solo ve el texto. Los dos brazos lo hacen igual (ninguno manda `thinking`).","La latencia es de pared desde este portátil, con A y B lanzados a la vez para cada turno; incluye la red doméstica y la carga del proveedor en ese momento. Los reintentos por 429 se excluyen de p50/p95 por llamada. Otros tres bancos compartían las claves a la vez."]}