A Training Step

A step is: make a prediction, see how wrong it was, work out which way each of the model's billions of dials should move, and then move them. Forward, backward, agree, update. The first two are arithmetic. The third is a conversation between every GPU in the job, and it is where the machine either earns its keep or does not.

Forward. The micro-batch flows through the layers, producing activations that have to be kept - or recomputed later - because the backward pass needs them.

Backward. Gradients propagate in reverse, costing roughly twice the arithmetic of the forward pass. The convention is that a step costs about 6 × parameters × tokens FLOPs: two for the forward, four for the backward.

Reduce. Every data-parallel replica has computed a gradient from different data, and they must be summed before anyone updates. This is the all-reduce of chapter 6 of Part II, and it is the only phase whose cost is set by the network rather than by the GPUs.

Update. The optimiser applies the averaged gradient. Cheap in FLOPs and expensive in memory traffic, since it touches every byte of optimiser state.

The phases overlap, and the overlapping is most of the engineering. Gradients for the last layers are ready while the backward pass is still working through the first, so the all-reduce is issued in buckets and runs underneath the remaining computation. Done well, the reduce is nearly invisible; done badly, or at a scale where the collective is slower than the compute it hides behind, it is exposed and shows up directly in step time.

This is why in-network reduction matters more than its raw bandwidth saving suggests. It does not merely make the collective faster - it makes the collective small enough to stay hidden.

Split the layers across GPUs and you have a production line. The problem with a production line is the start and the end: when the first item enters, everyone downstream is idle, and when the last one leaves, everyone upstream is. The longer the line, the more of the day is spent filling and draining it.

With pipeline parallelismPipeline parallelismConsecutive layers live on different GPUs, and micro-batches flow through them like a factory line. the layers are spread across p stages, and the batch is chopped into m micro-batches so that stages can work concurrently. But a stage cannot start until its predecessor has produced something, so there is a fill and a drain. The bubblePipeline bubbleThe idle time at the start and end of a pipelined step, while stages wait for work to reach them. is about (p − 1) / (m + p − 1) of the step.

That single expression explains a lot of practice. Deep pipelines need many micro-batches to amortise, which is why pipeline parallelism and large batches always appear together, and why a 16-stage pipeline on 16 micro-batches wastes about half the machine.

Model FLOPs UtilisationMFUModel FLOPs Utilisation - the share of a machine’s peak arithmetic that ends up doing the model’s own maths. is the honest measure. Take the arithmetic the model definition requires for a step, divide by the arithmetic the hardware could have performed in the time the step took, and everything else - communication, bubbles, recomputation, memory stalls, stragglers - lands in the gap.

Meta reports 38–43 %Model FLOPs Utilisation, Llama 3 405B BF16 MFU over the whole run. The single most useful number on this site for calibrating expectations: after every architectural decision Parts I and II describe, a real training job converts under half of peak FLOPS into model arithmetic. Meta AI BF16 MFU across the Llama 3 405B run on 16,384GPUs in the Llama 3 405B training run H100s at 700 W with 80 GB HBM3 - not NVL72 hardware. Cited for the scale at which the failure statistics were gathered. Meta AI H100s.^{Meta AI} That is a good result. It is also, stated plainly, under half: after every design decision in Parts I and II - the coherent domain, the copper spine, the lossless fabric, the in-network reduction - a state-of-the-art run converts a minority of peak FLOPS into model arithmetic.

Two things make MFU deliberately unforgiving, and both are correct. It counts recomputed activations once, as the model needs them, not twice as the hardware did them - so recomputation lowers MFU even when it is the decision that made the run fit. And it uses peak as the denominator, including a peak that assumes sparsity or a precision the run does not use. Hardware FLOPs Utilisation, which counts what the chip really executed, always reads higher, and is the less useful number: nobody wants a machine that is busy, they want one that is finishing.