The AI Inference Revolution: Why Running Models Is Becoming as Important as Training Them
Training makes the headlines. Inference is what actually shows up on the bill, every request, forever, and it now eats over 80% of AI GPU spend.
A new frontier model launch gets a press cycle: the training run, the cluster size, the benchmark scores. What doesn't make headlines is the number that actually shows up on an AI company's infrastructure bill every single month: at production scale, inference now accounts for more than 80% of total AI GPU spend. Training happens once, or periodically. Inference happens on every request, forever, for as long as the product exists.
The one sentence to remember
Training is a one-time capital cost that makes headlines. Inference is a recurring operational cost that makes or breaks the actual economics of running AI at scale, and it's where the real engineering work has quietly moved.
This is an infrastructure deep dive: what actually separates training from inference, why most GPUs serving inference today sit shockingly idle, and the specific mechanisms, batching, caching, quantization, routing, dedicated hardware, that are turning inference efficiency into the discipline that decides whether an AI product is profitable or not.
Training vs Inference
| Training | Inference | |
|---|---|---|
| What it does | Adjusts model weights against a fixed dataset | Runs a fixed, already-trained model against live, unpredictable requests |
| Frequency | Once, or periodically for a new version | Continuously, for as long as the product is live |
| Optimization goal | Maximum throughput over a large, known job | Latency and throughput simultaneously, under bursty, unpredictable demand |
| Failure mode if inefficient | The training run costs more and takes longer | Every single user request costs more and feels slower, all the time |
Why this distinction actually matters
A training cluster can run flat-out for weeks on a known workload. An inference fleet has to stay responsive to whatever traffic shows up, right now, while somehow also staying efficient during the much larger share of time when traffic is far below peak. That tension, staying fast under load and cheap when idle, is the entire inference engineering problem in one sentence.
GPU Utilization: The Uncomfortable Starting Number
According to Cast AI's 2026 State of Kubernetes Optimization Report, most production Kubernetes clusters run their GPUs at an average utilization of just 5%. Expensive, scarce hardware, sitting almost entirely idle, most of the time.
This is the single biggest lever in the entire topic
The most common cause is naive request handling: a GPU processes one request at a time, or waits for a fixed batch to fill before starting, leaving the hardware idle between requests instead of continuously fed. Fixing this one problem, covered next, took real deployments from roughly 22% utilization to 68%, more than triple, with no new hardware purchased at all.
Latency and Throughput
These pull in different directions, and conflating them leads to optimizing for the wrong thing.
| Metric | What it measures | Who cares |
|---|---|---|
| Latency | How long a single request takes, especially time to the first token and the gap between subsequent tokens | The user staring at the screen waiting for a response |
| Throughput | How many total tokens the system processes across all concurrent requests per second | The business, since it determines how many users one GPU can actually serve |
A system tuned purely for throughput can let individual requests wait in queue longer, hurting latency. A system tuned purely for latency, always processing requests immediately and alone, wastes the GPU's capacity to work on several requests at once. Production inference systems have to hold both simultaneously, not pick one.
Batch Inference vs Continuous Batching
Batch Inference
For work with no real-time constraint, bulk classification, offline summarization jobs, scoring a dataset overnight, requests get grouped into large batches and processed together purely for throughput. Nobody's waiting on any individual result, so there's no latency trade-off to manage at all.
Continuous Batching
Real-time serving can't wait for a batch to fill before starting, and it can't process one request at a time either without wasting most of the GPU's capacity. Continuous batching, the mechanism behind vLLM's PagedAttention and now the default in vLLM, SGLang, and TGI, solves this by admitting and evicting individual requests at the granularity of a single decoding step, so the GPU stays busy on a full, constantly-refreshing batch as requests arrive and finish at different times.
The actual measured impact
Continuous batching delivers two to five times the throughput of static batching, and is a direct, primary contributor to that jump from roughly 22% to 68% GPU utilization. If a production inference stack isn't using it, it's running at a fraction of what the same hardware could actually deliver.
KV Cache and GPU Memory
Every token a model generates attends back over every previous token in the sequence. Recomputing that attention from scratch for each new token would be prohibitively slow, so inference systems cache the key and value tensors from previous tokens, the KV cache, and reuse them.
The KV cache is the actual memory bottleneck, not the model weights
Model weights are a fixed size once loaded. The KV cache grows with context length and with the number of concurrent requests being served, and at long context lengths it can dominate GPU memory entirely. More concurrent long-context requests directly means less memory available to serve additional requests at all, a hard, physical trade-off, not a tuning knob.
The direct fix: quantizing the KV cache itself, typically to INT8 or FP8 instead of full precision, cuts its VRAM footprint by 30 to 50% at 32K-plus token context lengths, freeing that memory to serve meaningfully more concurrent requests on the same hardware.
Quantization
Quantization reduces the numerical precision used to store model weights and run computations, trading a small, often negligible amount of accuracy for a large gain in speed and memory efficiency.
This has become close to a free win
A common, current configuration, INT4 weights, FP8 KV cache, BF16 compute, delivers three to four times the throughput of a full FP16-everywhere baseline. On an H100, FP8 precision typically causes less than a 1% drop on standard accuracy benchmarks compared to BF16. At that ratio of throughput gained to accuracy given up, skipping quantization is difficult to justify for most production workloads.
Model Routing
Not every request needs your most capable, most expensive model. Model routing sends each request to whichever model, and whichever provider, is actually appropriate for that specific task's difficulty and cost sensitivity, rather than defaulting every request to the same one.
Inworld AI's Realtime Router is a real example of this pattern at scale, routing across more than 220 models spanning OpenAI, Anthropic, Google, Mistral, DeepSeek, and dedicated inference providers like Groq and Fireworks, matched to each scenario's actual requirements rather than a single fixed choice for every request.
Routing is a cost lever, not just a capability lever
A simple classification task doesn't need a frontier model's reasoning depth, and paying for it on every request compounds fast at scale. Routing the easy 80% of traffic to a smaller, cheaper, faster model while reserving the expensive one for genuinely hard requests is one of the more directly measurable cost optimizations available, without touching model quality on the requests that actually need it.
Cost Per Token
Frontier API pricing in 2026 spans a genuinely wide range, roughly $0.10 to $15 per million input tokens, and $0.40 to $60 per million output tokens, depending on model capability tier. That range alone is why every other technique in this article matters: the gap between the cheapest and most expensive viable option for a given task is often well over 10x.
Batching alone is a real, immediate cost lever
Continuous batching at even a modest batch size can push effective inference costs down to the $0.15 to $0.25 per million token range, a three to four times reduction, achieved purely through better request scheduling, with no additional hardware and no change to the model itself.
Dedicated Inference Infrastructure
At low, bursty volume, pay-per-token APIs are almost always the right call, no idle hardware, no operational burden. At sustained, high volume, the economics flip.
The break-even point is lower than most teams assume
Dedicated GPU inference infrastructure typically becomes mathematically cheaper than pay-per-token APIs once utilization reaches roughly 15 to 25%, well below full utilization. For any workload with predictable, sustained demand, owning or reserving inference capacity, rather than paying per token indefinitely, is often the better economic decision far earlier than intuition suggests.
Purpose-built inference hardware is also its own emerging category, distinct from general-purpose GPUs. Groq's LPU and Cerebras's WSE-3 wafer-scale chip are the two most prominent examples: Cerebras leads on raw throughput for small and mid-sized open models, roughly three to eight times faster depending on the benchmark, while Groq is generally cheaper per million tokens and known for especially consistent, low-latency serving. Neither is aimed at replacing general-purpose GPU clusters entirely; both compete specifically for the highest-value, highest-volume inference workloads where their specific advantage matters most.
Edge Inference
Running a smaller, quantized model directly on a user's own device, a phone, a laptop, an embedded system, trades raw capability for latency with no network round trip, genuine offline operation, and data that never has to leave the device at all. It's the right call for tasks a smaller model can actually handle well, and the wrong call for anything that genuinely needs a frontier model's full capability. For a practical, hands-on guide to actually running models locally, see AI Offline vs Online Models and How to Build Your Own Offline AI Application; this article's focus is the infrastructure economics, not the how-to.
Why This Adds Up to a Revolution
None of these techniques are individually revolutionary. Stacked together, they are:
| Technique | Typical gain |
|---|---|
| Continuous batching | 2 to 5x throughput; utilization from roughly 22% to 68% |
| Quantization (INT4/FP8 mix) | 3 to 4x throughput versus an FP16 baseline |
| KV cache quantization | 30 to 50% VRAM savings at long context |
| Model routing | Removes frontier-model cost from requests that never needed it |
These gains compound, they don't just add
A system running all of this together isn't just a bit more efficient than one running none of it. It's serving multiples more traffic on the same hardware footprint, which is exactly why inference efficiency, not the next training run, has become the place where AI infrastructure spend is actually won or lost at scale.
The Bottom Line
Training a frontier model is a singular, headline-grabbing event. Running one, correctly, efficiently, and affordably, for every request a real product generates, for as long as that product exists, is the sustained engineering discipline that decides whether an AI business's economics actually work. GPU utilization sitting at 5% by default, continuous batching more than tripling it, quantization multiplying throughput again on top of that, this is where the real infrastructure work in AI has moved, and it's exactly why industry attention is following it there.
The question worth asking about your own system
If you're running models in production, do you actually know your GPU utilization, whether continuous batching is enabled, and what quantization your KV cache is running at? If the honest answer is no to any of those, there's very likely a three-to-four-times efficiency gain sitting unclaimed in infrastructure you already own.
The model that gets trained once is the story. The infrastructure that runs it a billion times is the business.
Written by
Chetan Yamger
Cloud Engineer · AI Automation Architect · Modern Workplace Consultant
Cloud Engineer, AI Automation Architect, and Modern Workplace Consultant based in Amsterdam, Netherlands. Specializing in scalable, secure enterprise solutions with Microsoft Azure, Intune, PowerShell, and AI-driven automation using ChatGPT, Gemini, and modern LLM technologies.
Stay in the loop.
New articles, straight to you.
Deep-dive technical articles on Intune, PowerShell, and AI — no noise, no spam.
Discussion
Share your thoughts — your email stays private
Leave a comment