Tearing Inference Apart
I promised fewer numbers in this post, so let’s skip the 18 zeros and jump into it.
One of the many lessons you can learn from the TV show Seinfeld is that when you cannot buy more of something, you stop wasting what you have.
In this case, Monolithic inference wastefully forced two completely different workloads onto the same chip. Every generation request is actually two jobs posing as one:
- Prefill reads the prompt. It processes all input tokens in parallel, builds the model’s working memory (the KV cache), and produces the first token. It is compute-hungry and maxes out Tensor Cores.
- Decode writes the answer one token at a time. Every single token requires reading the entire accumulated KV cache from memory. It is sequential, memory-bandwidth hungry, and leaves Tensor Cores mostly idle.
Tune a server for prefill, and decode starves. Tune it for decode, and you pay for compute bandwidth that prefill barely touches. To escape that trap, operators stopped serving both from the same machine and decoupled them into specialized pools. DeepSeek runs prefill and decode in separate clusters at different levels of expert parallelism, and Meta runs disaggregated serving in production.
That split saved massive GPU capacity, but it introduced a new tax: it moved the model’s working memory off local silicon and onto the network.
Splitting the work
A 70B model using a 16-bit KV cache builds roughly 30 GB of cache for 100,000 tokens
When prefill and decode ran on the same machine, that data sat in local memory. Reading all 30 GB once at a current GPU’s published memory bandwidth takes about 4 milliseconds.
On separate machines, that same data has to cross an 800G Ethernet link. At nominal line rate, 30 gigabytes takes about 0.3 seconds.
In production, you never wait that full 300 milliseconds. Engines stream the cache layer by layer while the decode stage runs math, hiding the network behind compute. But the transfer is still on the wire, and it stays hidden only as long as the fabric doesn’t drop packets.
Caching out
Agents reread the same files and context repeatedly, so why not just cache it?
That works. On agent traces, adding a shared cache pool took the hit rate from 1.7% to 92.2%, quadrupled throughput, and slashed time-to-first-token by 46x.
But that cache does not live on one server; it is pooled across the cluster. A request landing on node A regularly fetches cache stored in node B’s memory.
So you didn’t eliminate the network. Instead of one big handoff, you now have constant reads against memory scattered across the entire fleet. The model’s working memory no longer lives inside a single box. It lives on the wire.
Cache-22
This creates a scheduling dilemma no algorithm can fully resolve:
- You want to route a request to the server that already holds its cached context, because a hit is free and a miss costs 30 GB across the wire.
- You also want to route it to the least loaded server, because queue delays destroy token generation speed.
NVIDIA’s Dynamo router scores each worker by how much of the cache it already has, then lets you weight that against load.
Prioritize cache affinity, and you get faster time-to-first-token at the risk of overloading servers. Prioritize load distribution, and you get higher generation throughput at the cost of frequent 30 GB network transfers. Every routing tweak directly alters the flow of packets across the datacenter fabric.
The problem is that the optimal balance changes continuously because inference traffic is completely unpredictable.
In a training cluster, one job runs for six weeks. Same model, same step, same collective, over and over like Groundhog Day. You can size the network before you buy the switches.
Inference clusters offer zero predictability:
Production data proves how unpredictable inference really is. FineServe’s analysis of 1.48 billion requests shows peak seconds absorbing up to 18% of hourly volume, while TraceLab found agent sessions dominated by autonomous tool loops that read 126,000 tokens of context to generate a mere 252.
A human kicks off the task; the agent generates the rest. Traffic is bursty, shaped by somebody’s orchestration code. Next year’s shape depends on software nobody has written yet.
Keep your cache
OpenAI does none of it.
Their inference chip, Jalapeño, runs prefill and decode on the same silicon without moving cache across the network. SemiAnalysis benchmarked it in August (paid), and it beat every NVIDIA, AMD, and Google part they have tested on tokens per megawatt.
Their design makes the opposite bet. Statically splitting hardware risks leaving decode chips idle whenever traffic shifts. Yet avoiding that split still took a private fabric of 2,048 chips across 16 racks: the architecture built to avoid the network is made of network hardware.
Google embraced the network. Their new chip uses a flat “Boardfly” fabric, cutting hops from 16 to 7 to attack compounding tail latency in agent loops, according to SemiAnalysis (paid).
Two companies that own their entire stack, going in opposite directions. One built a private network big enough to avoid the problem. The other redesigned its network around it.
If OpenAI solved this with monolithic custom chips, why shouldn’t everyone else do the same?
You can, right after you convince your CFO to fund a multi-billion-dollar custom chip team and a 16-rack island.
For the remaining 99.9% of the planet, you leverage a commercial Ethernet fabric.
The disaggregated era
Disaggregation solved the utilization problem by putting the network on the critical path. The cost did not vanish. It moved onto the AI network.
Part 3 is what happens when this traffic hits a real fabric: why you cannot tune it in advance, what tail latency costs, and how to operate a network whose shape you do not know yet.
