The OpenAI API is the fastest way to ship an LLM feature, and for many workloads it should stay exactly where it is. The reasons teams start looking elsewhere are specific: per-token pricing that scales linearly with usage until a high-volume workload dominates the cloud bill, data leaving your environment to a third party in regulated or sensitive contexts, model and version deprecations that force periodic prompt rework, and the strategic discomfort of a single-vendor dependency. Open-weight models you host yourself address all four, at the cost of taking on inference infrastructure. Whether that trade is worth it depends almost entirely on utilization.
When self-hosting actually saves money
Per-token API pricing has no fixed cost: you pay only for what you use, which is ideal for spiky or low-volume workloads. Self-hosting inverts that. You provision GPUs (or a managed inference endpoint) and pay for them whether or not they are busy, so the economics only work when you keep them reasonably utilized. The rule of thumb is that steady, high-volume, predictable traffic favors self-hosting, while bursty or experimental workloads favor staying on the API. Model the crossover with your real token volume in the calculator before committing, and be honest about GPU utilization rather than assuming a full load.
Llama vs Mistral as the open target
Llama (Meta) is the most widely deployed open-weight family, with broad tooling support, a range of sizes, and a large community, which makes it a safe default when you want the deepest ecosystem and the most published guidance. Mistral models are known for strong quality-to-size efficiency, which can mean lower GPU cost per request for a given quality bar, and Mistral also offers its own API if you want open weights without running the hardware yourself. For many teams the right move is to benchmark both on their own prompts rather than choosing on reputation.
The migration is an evaluation problem, not just a plumbing one
Technically, the path is well understood: stand up the model with an inference server such as vLLM or TGI, front it with an OpenAI-compatible gateway so your existing client code barely changes, migrate your prompts and function-calling definitions, and shift traffic gradually. The hard part is quality. Different models respond differently to the same prompt, so the real work is building an evaluation suite from your actual use cases and A/B testing the open model against the incumbent before you move production traffic. Treat “does it pass our evals” as the gate, not “does it return a response.”
The infrastructure nobody budgets for
Self-hosted inference is not one server with a GPU in it, and cost models that assume it is tend to be wrong by a wide margin.
Redundancy multiplies the hardware. A single inference node is a single point of failure for a production feature. Real deployments run at least two, usually across availability zones or sites, which means the GPU cost in your business case should be at minimum doubled before you compare it to an API bill that already includes redundancy.
Capacity is provisioned for peaks, and paid for at troughs. The API scales to your traffic automatically. Your hardware is sized for the busiest hour and idle the rest of the time. If your traffic has a pronounced daily or weekly shape, effective utilisation can be far below what a naive calculation assumes, and utilisation is the entire basis of the cost argument.
Someone operates it. Model updates, inference server upgrades, driver and CUDA version management, queue tuning, and capacity planning become your team’s work. This is ordinary infrastructure work and it is not free, so it belongs in the comparison as staff time rather than being quietly excluded.
Context length costs memory. Long-context requests consume GPU memory per concurrent request, so the model that fits comfortably in your benchmark may not fit at your real concurrency with your real prompt sizes. Test at production-like concurrency, not one request at a time.
The pattern that most often works economically is a split: keep the high-volume, well-defined, latency-tolerant workload on self-hosted open weights where utilisation is high, and leave the long tail of experimental and low-volume features on a hosted API. Chasing a hundred per cent migration is usually where the economics stop working.
Keeping a hosted fallback
Even a committed self-hosting decision benefits from not being absolute, and the cheapest insurance is architectural rather than contractual.
Route everything through an OpenAI-compatible gateway from the start, whether or not you have migrated anything yet. Once your application talks to a gateway rather than directly to a provider, switching models or providers becomes a configuration change instead of a code change, and you can shift traffic incrementally, run a percentage-based comparison in production, or fail over to a hosted endpoint if your inference cluster has a bad day.
That same indirection is what stops this migration from becoming the next lock-in. Open weights protect you from a provider changing its terms, but only if your application is not written against one provider’s particular API surface. The gateway is what makes the independence real.
It is also worth keeping a modest hosted-API budget alive after migration for evaluation purposes. Comparing your self-hosted model against the current frontier every few months is how you find out whether the trade-off you made a year ago still holds, and per-token pricing makes that an inexpensive habit. Each path below opens to a full cost model, a phase-by-phase plan, and an in-depth guide.