What breaks when you switch models?

We ran one old model and five replacements on the same 320 production-style items, at temperature 0, and compared them item by item. The newer models were better on average, and every one of them still broke something that used to work.

Figure 1. For each replacement, items the old model handled correctly that it got wrong (broke, amber) and items the old model got wrong that it handled correctly (fixed, white), summed over all 320 items in the five sets. A net score hides this: gpt-oss-120b gains 21 and loses 33.

Version 1.1. v1.0 (12 Nov 2026) compared four replacements. v1.1 adds gpt-oss-120b, an open-weight model, with the second judge re-run on all 6 models; the other models' replies and scores are unchanged.

Why now

Several model retirements land in the next ten weeks. The dates below are copied from the providers' deprecation pages on 12 November 2026:

• Claude Sonnet 4.5 (claude-sonnet-4-5-20250929) retires November 30, 2026 (Anthropic model deprecations page).
• From December 1, 2026, gemini-2.5-flash calls route to gemini-3.5-flash-lite and gemini-2.5-pro calls to gemini-3.8-flash (Gemini API deprecations page). Nothing errors; the model behind the same name changes.
• OpenAI removes gpt-5, gpt-5-mini, gpt-5-nano and o3 from the API on December 11, 2026, and the GPT-4o, GPT-4o mini, GPT-4.1 and GPT-4.1 mini snapshots on January 14, 2027; fine-tunes on them go with them (OpenAI deprecations page).

A retirement forces a switch on a fixed date. The question a team has to answer before that date is not "is the new model better?" but "which of our cases does it get wrong that the old one got right?" This study measures exactly that, on a small, public set you can rerun.

1. Results

1.1 Summary

Table 1. Items handled correctly out of the items in each set. Roles: Gemini 3 Flash is the old model (a stand-in); Gemini 3.8 Flash is the same provider's newer Flash; the two Flash-Lite models are its cheaper tiers; Claude Sonnet 5.5 is another provider; gpt-oss-120b is an open-weight model. “$ / 1k items” is the measured cost of 1,000 items averaged over all five sets at list prices; latency is the median over all 320 calls through our gateway (the support-only medians quoted above are lower or higher per model). Full confidence intervals are in the results CSV.

Figure 2. Items each model handled wrongly, by task; brighter is worse. Extraction is near-saturated; the differences sit in the support answers and the tool calls.

1.2 What broke and what got fixed, item by item

Figure 3. For each candidate and task: items the old model got right and the candidate got wrong (−), and the reverse (+). Brighter cells broke more.

Table 2. Paired comparison with the old model on the same items (main extraction set omitted: no item changed for any model; the hard extraction set is listed only where an item changed). The exact McNemar test uses only the items that changed; p < 0.05 is highlighted. With 40–80 items, only a few differences are clearly beyond chance.

1.3 Support answers grounded in a policy

Table 3. 80 customer e-mails to a fictional outdoor store, each with the 31-rule policy, the true order and charge records, and a gold resolution written by us. “Correct” = resolved correctly or handed off when policy requires it. “Invented policy” is the primary judge's flag for any rule, time frame, fee or action not in the policy or records (claude-opus-5-5-high); the second judge (gemini-4-carbon-high) flagged 3, 0, 1, 2, 0, 10 in the same model order. A code-only check for any $, % or time figure absent from the policy, e-mail and records flagged 1, 0, 1, 0, 1, 0 replies in the same order.

The old model's main weakness was handing routine cases to a human (13 of 58 resolvable e-mails), often with a made-up reason. Example (S-T-1229): “Account changes like updating your e-mail address must be handled by a team member.” The policy says customers change their e-mail themselves under Account > Profile > Email. The two stronger replacements mostly stopped doing this; the two Flash-Lite models did not. gpt-oss-120b almost never handed off needlessly (1), but it resolved 16 e-mails wrongly, twice the old model's 8.

The regressions are the part a migration has to catch. Gemini 3.8 Flash resolved a $229 refund on its own (the policy requires a person to approve refunds above $150; the old model handed it off), and told a customer that promo codes only work on “full-price items”, a condition the policy does not state. Claude Sonnet 5.5 handed off an $89 defect refund the bot was allowed to resolve. Each is one e-mail in 80; at 10,000 e-mails a month the same rate is roughly 125.

1.4 Tool calls with a strict JSON schema

Table 4. Main set: 80 requests to a support agent with 7 tools (strict JSON Schema, enums, patterns, no extra properties), 10 of which need no tool. Hard set: 40 requests where the customer has two orders with the same item, returns two of three items, mistypes the order number, sees three charges (two posted, one pending), or gives an international address with a country name instead of an ISO code. Scored on exact arguments; no judge. “Schema violation” counts calls whose arguments break the declared schema, over both sets.

No model produced invalid JSON in any of the 720 tool requests. Only gpt-oss-120b broke the schema: 16 calls sent a country name where the country field requires a two-letter ISO code, which the gateway passed through. The other failures were semantic: the right tool with the wrong value. The old model and both Flash-Lite models refunded a pending authorization (2, 2 and 2 of 3 trap requests), and passed the customer's mistyped order number to get_order_status in 2, 3 and 2 of 8 cases, a lookup that would fail or hit another customer's order. Picking the later posted charge out of three went wrong 2, 3 and 2 times out of 8. Gemini 3.8 Flash made none of these errors; Claude Sonnet 5.5 missed one call: it noticed the mistyped order number and asked the customer to confirm the real one, arguably the safest reply, but not the tool call the set expects.

1.5 Structured extraction from messy invoices

Table 5. 80 + 40 synthetic supplier invoices in 10 layouts (US e-mail paste, German invoice with 1.234,56 € amounts, OCR'd UK table, forwarded statement with a previous balance, one-line compact; credit note, net/gross VAT, dd/mm dates, two pages with a carried-forward subtotal, a corrected PO in an e-mail thread) into 11 fields with nulls for absent fields. Exact field match after normalisation; no judge. “All fields” = all 11 fields of a document right.

Five of the six models got every field right. We wrote the hard set after the main set came back at 100%, before running any model on it; for those models it also came back at 100%. gpt-oss-120b missed 2 hard invoices (the invoice number on 1 and the currency on 2). For extraction of this kind the migration risk is low, and a short regression set is enough to confirm it.

1.6 Cost and latency

Figure 4. Cost per 1,000 support answers (measured tokens × list price) against support answers handled correctly. Amber: the old model.

Table 6. Prices from the KH Labs price table, read from the providers' pricing pages on 2026-11-12 (Google, Anthropic, Groq). gpt-oss-120b is open-weight and sold by several hosts; we price it at Groq's list price ($0.15 / $0.60 per million input/output tokens). Tokens are the gateway's usage counts, including hidden reasoning tokens (“Reasoning” = average reasoning tokens per support answer; our gateway reported none for gpt-oss-120b). Gemini 3 Flash is priced as Gemini 3 Flash (preview), the price-table entry for that id. Latency is measured through our gateway, not the providers' APIs directly.

1.7 The noise floor: temperature 0 is not deterministic

Table 7. “Same” = output identical to the first run; “flip” = the hand-off decision changed; “other score changes” counts tool and extraction items whose score changed. The first 20 items of each main set sent a second time to the same model with the same prompt, temperature 0 and the same seed (not repeated for gpt-oss-120b, added in v1.1). Support replies were almost never word-for-word identical (at most 1/20), and the hand-off decision flipped on up to 3/20 e-mails. A difference of one to three items between two models is within this noise; that is why we report paired counts and a test, not a single score.

2. Method

Models. Only models available through our OpenAI-compatible gateway on 12 November 2026. The “old” model is gemini-3-flash, the oldest Flash model available to us; it is a stand-in, not a model with an announced retirement. Candidates: gemini-3.8-flash-high (newer Flash from the same provider, with reasoning on), gemini-3.5-flash-lite (the cheaper tier Google routes gemini-2.5-flash to from 1 December), gemini-3.1-flash-lite (the cheapest tier), claude-sonnet-5-5-high (a different provider) and gpt-oss-120b-medium (open-weight, added in v1.1). Every model received the identical prompt; nothing was tuned per model. Models served through vendors’ CLI clients (the Codex CLI default model, Claude Haiku 5.5, Claude Fable 5.2) are not reported: they did not finish every set or lack the judge labels the report requires.

Items. Support: 80 e-mails sampled round-robin over 35 scenarios from the holdout split of our Alder & Ash support demo, with its policy, records, gold resolution and code checks. Tools and extraction: generated deterministically (seed 20261112) from the same fictional store's catalogue and from synthetic supplier invoices. All companies, people, orders and invoices are fictional.

Runs. 1,920 scored calls for the six reported models on 12 November 2026, temperature 0, seed 7 (accepted by every model; whether it is honoured is not visible), tools passed as OpenAI tools, max 4,000 output tokens, no errors after retries. Every request and response is cached by request hash, so a rerun makes no calls.

Scoring. Tools and extraction are scored by code only. Support uses the code checks from the demo (stale facts, required facts, no refund promise when ineligible, no foreign order ids) plus an LLM judge with the gold resolution and full policy (claude-opus-5-5-high); the hand-off decision is read from a [HANDOFF] tag, not judged. A second judge from another provider (gemini-4-carbon-high) labelled all 480 support replies: 95.0% agreement on pass/fail (κ = 0.82) and 96.7% on the invented-policy flag (κ = 0.65, moderate to substantial; that flag is the least certain number in this report). Rates carry Wilson 95% intervals in the CSV; paired differences use the exact McNemar test.

3. What the numbers cover

• The old model is a stand-in. None of the retiring models above was available through our gateway, so we cannot say what breaks when you leave GPT-4o, gpt-5-mini or Claude Sonnet 4.5 specifically. The method transfers; these numbers do not.
• Small, synthetic sets. 80 support e-mails, 120 tool requests and 120 invoices, all invented. Real traffic is messier and has a different mix; with 40–80 items, differences of a few items are not significant (Table 2) and fall inside the rerun noise (Table 7).
• One support domain, one policy, English only. Gold answers are written by us; a different policy author could draw some lines (e.g. when a hand-off is needed) differently.
• Judge dependence. Support correctness for answered e-mails depends on an LLM judge. Two judges from two providers agree on pass/fail at κ = 0.82, but on the invented-policy flag only at κ = 0.65. We did not hand-label these replies.
• One gateway, one day, one sample per item. All models were called through one OpenAI-compatible gateway; provider-native APIs may report different token counts and latency, and may handle sampling settings such as temperature differently. The gateway did not enforce the tool schema for gpt-oss-120b; a host that does would turn its schema violations into errors. Models behind the same name also change over time.
• The hard sets were added after the main sets saturated. They were written before any model saw them and are reported in full, but they were not part of the original plan. One gold label (credit notes print no due date) was corrected after the first scoring pass; the model outputs were not re-run or changed.
• Default prompts. We did not re-tune prompts per model. In a real migration some regressions are fixed by a prompt change; this study measures the drop-in switch.

4. How to run it on your own traffic

1. Export 100–300 real inputs from your logs for each use case, with the old model's output. Keep the hard ones: refunds near a limit, hand-off rules, ids typed by customers.
2. Write down what “right” means per item: the expected tool and arguments, the fields, or the outcome (resolve or hand off) plus the facts a correct answer must contain. Code checks first; an LLM judge only where text quality matters, validated against a second judge or a human sample.
3. Run the old model twice to measure your noise floor, then each candidate once, with the same prompts.
4. Compare item by item, not by average. List every item that broke, read them, and decide which ones you can fix with a prompt change and which ones block the switch.
5. Price it from the usage logs, including reasoning tokens, and check latency at your real prompt sizes.
6. Keep the set. The next retirement will come; rerunning a cached test set costs minutes.

The code does all of this for any OpenAI-compatible endpoint: replace the files in data/, set your model ids in run.py, and run generate, judge, analyze (see the README in the zip).

Appendix

Data. r4_model_task.csv (model × task with Wilson intervals, cost, latency, tokens), r4_items.csv (every scored item of the six reported models), r4_regressions.csv (named regressions vs the old model). Code and raw outputs: migrationbench-v1.1.zip (items, prompts, judge prompt, every response and judge label of the reported models, cached raw API responses; Python standard library only).

Cite this report