Chapter 08
Software & Inference
Serving a language model is two different jobs wearing one coat. They have opposite bottlenecks, they want different hardware ratios, and for years they were forced to share a GPU anyway. A 72-GPU coherent domain is what makes separating them practical.
Two phases, opposite bottlenecks
When you send a prompt to a language model, two things happen in sequence.
First it reads your whole prompt at once. That is a large, dense pile of arithmetic and it keeps the chip's maths units busy — this phase is limited by how fast the GPU can compute. Then it writes the answer one word at a time, and each word requires re-reading the model's entire set of weights to produce a tiny amount of arithmetic. That phase is limited by how fast the GPU can read memory, not by how fast it can compute.
Those are two different machines' worth of requirements. Running both on the same GPU means neither gets what it wants.
Prefill processes every prompt token in parallel: large GEMMs, high arithmetic intensity, tensor-core-bound. It sets time-to-first-token. Decode generates one token at a time, re-reading the full weight set and the growing KV cache per step. Its arithmetic intensity is close to one, so it is memory-bandwidth-bound outright, and its only real lever is batching more requests together to amortise those weight reads.
Co-locating them forces a compromise on every axis: batch composition, memory budget, parallelism width, and scheduling. Worse, a prefill burst stalls the decode steps sharing the GPU, which shows up directly as inter-token latency jitter — the thing interactive serving is judged on.
This figure is interactive and needs JavaScript. The prose around it states every number it shows.
Disaggregated serving
NVIDIA Dynamo runs the two phases on separate GPU pools and streams the KV cache between them.NVIDIA Each pool is then free to be configured for its own bottleneck: prefill for throughput, decode for wide expert parallelism and large batches. On MLPerf Inference v5.1, disaggregated serving on GB200 NVL72 reached ~1.5× throughputDisaggregated vs aggregated serving MLPerf Inference v5.1, Llama 3.1 405B interactive. MLCommons · NVIDIA · NVIDIA Technical Blog over aggregated serving for Llama 3.1 405B interactive.MLCommons,NVIDIA
The reason this needs an NVL72 rather than any cluster is the KV cache transfer plus the expert all-to-all. Moving a long-context KV cache from a prefill GPU to a decode GPU is a large, latency-sensitive transfer sitting directly in the request path. Over InfiniBand it is a cost you schedule around; over 130 TB/sAll-to-all NVLink bandwidth in-rack NVIDIA of in-rack NVLink it is close to free.
Wide expert parallelism is the second-order effect and arguably the larger one. Spreading a mixture-of-experts model's experts across many GPUs cuts the weight memory each GPU holds, which frees HBM for KV cache and larger batches, which is precisely what a memory-bound decode phase needs.NVIDIA Technical Blog The cost is an all-to-all exchange on every layer whose size grows with the number of participating GPUs.
That exchange is what makes wide EP unaffordable on a scale-out fabric and affordable here: it stays inside the NVLink domain, where per-GPU egress is 900 GB/s rather than 50 or 100. The 72-GPU domain is not incidental to disaggregated MoE serving; it is the precondition for it.
The rest of the software story
Three pieces of plumbing matter enough to name.
- Topology awareness. NCCL and CUDA know the NVLink topology and pick collective algorithms accordingly — including using SHARP in-network reduction where it applies. Nothing about the fabric is exposed as something you tune by hand.
- Domain boundaries. MNNVL and the IMEX service let GPU memory be exported across OS domains; Kubernetes models this with a ComputeDomain resource via the GPU Operator and DRA driver.NVIDIA Docs,NVIDIA Docs IMEX itself does not depend on CUDA — it brokers the export/import handshake between compute nodes over TCP and gRPC. The NVLink domain and the scheduling domain are separate concepts.
- Partitioning. MIG still slices a single GPU into isolated instances — relevant for inference tenancy, irrelevant to the training story.
Measured results
| Disaggregated vs aggregated serving | ~1.5× throughput | MLPerf Inference v5.1, Llama 3.1 405B interactive. |
|---|---|---|
| Llama 3.1 405B training time | 27.3 min | MLPerf Training v5.0, 4 June 2025: 27.33 minutes on 2,496 Blackwell GPUs across 39 racks running 64 active GPUs each — not fully populated 72-GPU racks, which is why 2,496 does not divide by 72. CoreWeave puts an equivalent H100 setup at around 156 racks, assuming 32 GPUs per rack. A later round reached about 10 minutes on more than 5,000 Blackwell GPUs. |
| Blackwell NVFP4 training speedup vs Hopper FP8 | up to 3.2× | MLPerf Training v5.1, Llama 3.1 405B, at the same GPU count. |
| GB300 training speedup | 4.2× vs Hopper, 1.9× vs GB200 | At 512-GPU scale. |
| Real-time trillion-parameter LLM inference | 30× | NVIDIA’s measured configuration: TTL = 50 ms, FTL = 5 s, 32,768 input / 1,024 output tokens, GPT-MoE-1.8T, comparing 64 Hopper GPUs over InfiniBand against 32 Blackwell GPUs in an NVL72. Not a like-for-like GPU count. |
| Mixture-of-experts performance | 10× | Same measured configuration as the 30× figure. |
Why reliability shows up as a serving concern
Blackwell's RAS engine runs built-in self-tests and predicts failures so that board replacement becomes a scheduled event. The reason this belongs in a chapter on inference rather than one on hardware: the failure domain is the NVLink domain. Losing a compute tray removes four GPUs from a 72-GPU fabric, and any job whose collectives span it must be reformed. Designing serving topologies that degrade rather than stop — and knowing which tray a replica lives on — is an operational discipline this density forces on you.