When Should Your Fallback Model Become the Primary? A Break-Even Rule With Real Prices

You added a fallback so the app wouldn't break. Then the invoice came.

A fallback is one of the first reliability features you add. The cheap model handles most requests. When it fails, times out, or returns something your code rejects, a stronger model takes over. The user never sees the failure. You see it later, on the bill.

Then the obvious question shows up: the fallback fires a lot, so should you just make the stronger model the primary? Or try a different cheap model? You can answer that with two numbers: what one attempt costs on each model, and how often the fallback fires. This post works through both with current list prices, then shows how to run the same check on your own workload.

Every example below uses the same workload: 3,000 input tokens and 600 output tokens per attempt, 100,000 tasks a month, Standard pricing, no batch discount, and no prompt caching unless a section says otherwise. Prices come from the AICostBudget dataset, which links every row to the provider's own pricing page. The table at the top of this page shows each source and the date it was last checked.

Cost per attempt is (3,000 × input price + 600 × output price) ÷ 1,000,000. If your token counts differ, look the model up on its model pricing page or in the full pricing comparison and redo the multiplication with your numbers.

A 5% fallback rate can double your bill

Take the common setup: GPT-6 Luna as the primary, Claude Sonnet 5.5 as the fallback. A Luna attempt costs $0.0006. A Sonnet 5.5 attempt costs $0.012, which is 20 times more. When Luna's answer fails your check, you have already paid for it. Then you pay for Sonnet on top.

So each fallback adds the price of 20 Luna attempts. At a 5% fallback rate, the bill doubles. At 10%, it triples.

Now look at the bottom bar. Running Sonnet 5.5 for everything costs $1,200. The Luna route with a 50% fallback rate still costs $660. The two routes only cost the same at a 95% fallback rate. The general version of that rule is short:

break-even fallback rate = 1 − (primary cost per attempt ÷ fallback cost per attempt)

Luna → Sonnet 5.5:  1 − (0.0006 ÷ 0.012) = 0.95  →  95%

If your fallback fires 30% of the time and you are tempted to go all-in on the stronger model, the math says you would pay $1,200 a month instead of $420. A high fallback rate is a signal to look at the primary model. It is almost never a reason to make the expensive model do all the work.

The switch rule: how far must the fallback rate drop?

The more useful question is whether a different cheap primary would fail less often. A stronger primary costs more per attempt, but every fallback it avoids saves the full price of a fallback call. The switch pays off when the savings beat the extra cost:

switch pays off when the fallback rate drops by more than
(new primary cost − old primary cost) ÷ fallback cost

Luna → Gemini 3.8 Flash, fallback Sonnet 5.5:
(0.0045 − 0.0006) ÷ 0.012 = 0.325  →  32.5 percentage points

• Claude Haiku 5.5 has the same list price as GPT-6 Luna. If it fails your checks less often on your data, the switch saves money from the first request. This one is worth testing first.
• DeepSeek V4.1 Flash needs to cut the fallback rate by 8.5 points, for example from 20% down to 11.5%. That is a realistic bar if Luna struggles with your task.
• Gemini 3.8 Flash needs 32.5 points. That only works if Luna fails a lot, say 40% of the time, and Gemini brings it under 7.5%.
• GPT-6.1 Sol costs the same as the fallback. As a primary it only makes sense if you drop the fallback entirely, which brings you back to the 95% break-even above.

List prices assume both models use the same number of tokens. They often don't. A model that writes 900 output tokens where another writes 600 costs more than its rate card suggests. Run 200 to 500 real tasks through each candidate, record the fallback rate and the actual token counts from the usage field, and then apply the rule. The head-to-head pages for Claude Haiku 5.5 vs GPT-6 Luna, DeepSeek Flash vs GPT-6 Luna, and Gemini 3.8 Flash vs GPT-6 Luna show the current prices side by side.

Rate-limit failover is a different bill

Everything above is about quality fallbacks: the primary answered, and your code rejected the answer. Availability fallbacks are different. A 429, a 5xx, or a timeout often means the primary never produced a full answer, and the fallback is usually a model in the same price tier from another provider.

Gateways handle this in slightly different ways. OpenRouter's documentation says rate limits, downtime, context-length errors, and moderation flags can all trigger a fallback, and that requests are priced using the model that was ultimately used. LiteLLM's reliability docs describe retrying the same model group num_retries times first, then falling back to other groups in list order. Know which one you run, because retries before the fallback add attempts to every failed task.

One rule from OpenAI is worth remembering: unsuccessful requests count toward your per-minute rate limit. Hammering the same endpoint after a 429 does not buy you capacity. It burns more of it.

The cost people miss is the prompt cache. A cache lives with the provider that built it. Say your GPT-6.1 Sol requests reuse a long system prompt, so 2,500 of the 3,000 input tokens are cache hits at $0.10 per million. That attempt costs $0.00725. When the same request fails over to Claude Sonnet 5.5, the cache is cold, and the attempt costs $0.012. Same list price, 66% more per request.

Put that in monthly terms. If 3% of 100,000 tasks fail over during provider incidents, the cold cache adds 3,000 × $0.00475 = $14.25 to a $725 month. That is cheap insurance. Keep the failover.

What is not cheap is retries stacking on top of each other. AWS's Builders' Library article on timeouts, retries, and backoff with jitter walks through five service layers that each retry three times: the bottom layer sees 243 times the load. Their advice is to retry at a single point in the stack. Google's SRE book chapter on handling overload adds two limits: at most three attempts per request, and a client-wide retry budget that only allows retries while they stay below 10% of requests. Google reports that budget keeps load growth to about 1.1x instead of nearly 3x.

In an LLM app, the layers are your SDK, your gateway, your own retry loop, and sometimes an agent framework on top. If each one retries, a single bad minute at the provider multiplies into a very expensive one. Pick one layer to own retries and turn the others off.

Run your own numbers in about ten minutes

The examples above use round assumptions. Your token counts, fallback rate, and models are different. Here is the order that gets you to a decision fastest.

• Step 1. Pull real numbers from your logs: average input and output tokens per attempt (from the usage field of each response), tasks per month, and the share of tasks that reached the fallback in the last 30 days.
• Step 2. Price the route you run today in the LLM Retry & Fallback Cost Calculator. Set Primary model and Fallback model, Monthly tasks, and the two token fields. If each failing task gets exactly one fallback attempt, enter your fallback rate as Average extra retries per task (5% becomes 0.05) and set Share of retry attempts sent to fallback to 100%.
• Step 3. Price the candidate in the Model Switch Cost Calculator. Put your current primary as Current model and the candidate as Target model, with the same tokens and Requests per month. That gives you the per-attempt cost gap at current list prices.
• Step 4. Divide that gap by the fallback model's cost per attempt. That is the fallback-rate drop the candidate has to deliver. Compare it with what you measured on your 200 to 500 test tasks.

If you want the longer explanation of how retries, fallback calls, and unresolved tasks add up, read the cost of retries and fallbacks. For comparing routes by completed work rather than requests, see cost per successful task.

What this math leaves out

• Quality. A cheaper route that passes your check but annoys users is not cheaper. Define what counts as an accepted answer before you compare models.
• Output length. Models answer the same prompt with different token counts. Measure it; don't assume it.
• Discounts. Batch processing and prompt caching change the per-attempt price on both sides. Use the same assumptions for every model you compare.
• Unresolved tasks. Some tasks fail even after the fallback. They still cost money and produce nothing, which the retry calculator tracks as the final unresolved rate.
• Price changes. Every number in this post is a list price checked in October 2026. Providers change prices, so recheck before you commit.

Price changes are logged with dates and sources on the model price monitor, and the methodology page explains how each price is verified against the provider's own page. Your provider's invoice is still the final word on what you paid.

FAQ