Key takeaways
- Published break-even points for self-hosting differ by roughly 50x, because each one hides different assumptions about utilisation, staffing and which API it compares against.
- A rented GPU bills identically at full load and at idle. At 10% utilisation the effective cost per token is ten times the number on the invoice divided by capacity.
- All-in cost is reported at 3x to 5x raw GPU rental once engineering time is counted, with one mid-level MLOps engineer covering only 4 to 6 GPUs.
- Reported break-even against hosted open-weight APIs sits at 15M to 20M tokens per day, sustained. Bursty traffic at that daily average does not qualify.
- The question worth asking is not whether self-hosting is cheaper. It is what your utilisation would actually be, measured over a week that includes a weekend.
Does ChatGPT recommend your competitor instead of you?
Check how AI assistants describe your company, and whose name they give when someone asks for a recommendation in your category. Run an audit today, from $19.
Search for the point at which self-hosting an open-weight model beats an API and you will find credible-looking answers that disagree by a factor of fifty. One analysis says a few million tokens a day. Another says you need three years of continuous heavy load.
Both are arithmetically fine. They are answering different questions, and the difference is almost entirely a variable neither of them puts on the page.
Why do the published estimates disagree so badly?
Because three assumptions sit underneath every model and are rarely stated.
Which API is the baseline. Comparing self-hosting against frontier API pricing produces a flattering result. Comparing it against a hosted version of the same open-weight model, which is what your workload could actually use, produces a much harsher one. Most published estimates quietly do the former.
What utilisation is assumed. This is the big one, and it gets its own section.
Whether people are in the model. Reported all-in cost runs three to five times raw GPU rental once engineering time, on-call and headroom are counted. An estimate built from hourly GPU prices alone is off by that multiple before it starts.
What does utilisation actually do to the number?
It moves it more than any other input, and it is the one you cannot know in advance.
A rented GPU bills identically whether it is serving eight thousand tokens a second or nothing at all. So the effective cost per token is the rental divided by what you actually pushed through it, not by what it could theoretically do.
| Average utilisation | Effective cost per token | What this looks like in practice |
|---|---|---|
| 90% | 1.1x | Continuous batch workload, no daily cycle |
| 50% | 2x | Business-hours traffic with warm capacity overnight |
| 25% | 4x | Interactive product with a weekday peak |
| 10% | 10x | Agent fleet with bursty, task-driven load |
The last row is where most agent workloads sit, and it is the row that turns a favourable break-even into an unfavourable one without changing anything else in the model.
Why agent traffic is the worst case for utilisation
Agent load is task-driven, and tasks arrive when people or schedules create them. That produces sharp peaks with long troughs, which is exactly the pattern that fixed capacity handles worst. You size for the peak, you pay for the peak, and you use the trough.
An API charges you for the trough at zero. That is not a small structural advantage, and no amount of GPU price shopping closes it.
What does the honest comparison look like?
Four numbers, and you need all four before the question is answerable.
| Input | How to get it | Why estimates omit it |
|---|---|---|
| Tokens per day, measured over a full week | Your existing API bill, daily | Averages hide the weekend |
| Peak to trough ratio | Same data, hourly | It is the number that decides capacity |
| Baseline API you would otherwise use | Hosted version of the same open-weight model | Frontier pricing makes the case look better |
| Engineering cost at 4 to 6 GPUs per person | Your own loaded salary figures | It moves the answer by 3x to 5x |
If you cannot produce the first two from your current telemetry, that is the finding. A team that does not know its own peak to trough ratio is not in a position to buy fixed capacity, whatever the spreadsheet says.
Where does self-hosting genuinely win?
Three situations, none of which is “we want to save money in general”.
- Sustained high volume with a flat profile. Batch classification, enrichment pipelines, anything that can be queued and smoothed. This is the case the favourable estimates are describing.
- Data that cannot leave. When the constraint is regulatory rather than financial, the comparison is not against an API. It is against not shipping.
- A small model doing well-specified work. The hybrid pattern most teams land on: local capacity for the high-volume simple steps, a frontier API for the reasoning that decides the outcome.
The third is the one worth planning for, because it does not require the break-even argument to be won. It requires only that some of your steps are boring, which they are.
The question for your next infrastructure review
Not “should we self-host”. Ask: what was our average GPU utilisation going to be, measured from last month’s actual traffic including nights and weekends? Multiply the rental by the reciprocal of that number before anyone opens a comparison spreadsheet.
Frequently asked questions
When does self-hosting an LLM become cheaper than an API?
Reported break-even against hosted open-weight APIs runs from 15M to 20M tokens per day, and only with sustained rather than bursty load. Against the very cheapest hosted options, published analyses put parity at around 36 months of heavy continuous usage.
Why do self-hosting cost estimates vary so much?
Because they compare against different baselines and assume different utilisation. An estimate benchmarked against frontier API pricing at 90% GPU utilisation and one benchmarked against a budget open-weight API at 20% utilisation will differ by more than an order of magnitude while both being arithmetically correct.
What is the real cost of running your own inference?
Reported all-in cost is three to five times raw GPU rental. The difference is engineering time, on-call coverage, capacity headroom for peaks, and the throughput you lose to imperfect batching. One mid-level MLOps engineer is reported to cover four to six GPUs.
Should you use a hybrid of self-hosted and API models?
It is the pattern most teams settle on. A small self-hosted model handles high-volume, well-specified steps where quality requirements are modest, and a frontier API handles the reasoning that decides the outcome. Route per step rather than per workload.
Method and sources
- 2026 total cost of ownership analyses of local deployment against cloud APIs, reporting break-even at 15M to 20M tokens per day and parity against the cheapest hosted options at around 36 months of sustained heavy usage.
- 2026 write-ups reporting all-in cost at three to five times raw GPU rental, and staffing ratios of one mid-level MLOps engineer per four to six GPUs.
- Reported observation that published break-even estimates disagree by roughly 50x because of undisclosed assumptions.
- GPU lead time and allocation figures from 2026 infrastructure analyses, consulted for context on why capacity cannot simply be added when utilisation rises.
- The utilisation arithmetic in this article is derived from the stated premise that a rented GPU bills the same whether loaded or idle. It is arithmetic, not measurement, and is labelled as such in the table.