Something Is Always Broken

A single GPU is very reliable - it will run for years. Put sixteen thousand of them in one job and the arithmetic turns against you: something that happens once per component per few years happens to some component every few hours. Nothing has got worse. There is just more of it.

So a big training run is not a computation that occasionally fails. It is a computation that is always in the process of failing somewhere, and the job of the software around it is to make sure that costs minutes rather than days.

Over a 54-day snapshot of the Llama 3 405B run on 16,384GPUs in the Llama 3 405B training run H100s at 700 W with 80 GB HBM3 - not NVL72 hardware. Cited for the scale at which the failure statistics were gathered. Meta AI H100s, Meta logged 466 job interruptions. Forty-seven were planned; the other 419Unexpected interruptions in 54 days Out of 466 total; the other 47 were planned. Over a 54-day snapshot, that is a mean time between unexpected failures of roughly 3.1 hours. Meta AI were not. That works out to a mean time between unexpected failures of ~3.1 hMean time between unexpected failures Derived: 54 days over 419 unexpected interruptions. Meta does not state this figure directly; it follows from two numbers that are stated. Meta AI .^{Meta AI}

The breakdown is the useful part. 30.1 %Interruptions traced to a faulty GPU The largest single category. HBM3 failures add a further 17.2%, so roughly half of all unexpected interruptions are the accelerator or its memory. Meta AI of interruptions were a faulty GPU and a further 17.2% were HBM3 - so roughly half were the accelerator or its memory. Software bugs accounted for 12.9%, network switches and cables 8.4%, host maintenance 7.6%, GPU SRAM 4.5% and the GPU's system processor 4.1%.

Two things about that distribution are worth dwelling on. The first is that it is dominated by the components you have the most of, which is the whole lesson: reliability at scale is a counting problem before it is an engineering one. The second is that network faults are only 8.4% - a reminder that the elaborate fabric of Part II is not where the fragility lives, despite being the part that looks most fragile.

Chapter 2 of Part I makes the point that a compute tray is front-serviceable and blind-mates its liquid connections, so replacing one takes minutes and never breaks the coolant loop. That is the replaceable unit. It is not the unit within which a fault is contained.

Losing one tray removes four GPUs from a 72-GPU NVLink domain, and every collective of every job spanning that domain has to be reformed. So the blast radius of a single failed component is the rack - the same coherence that makes the rack one accelerator makes it one failure domain. Blackwell's RAS engineRAS engineReliability, Availability, Serviceability - on-die logic that runs self-tests and predicts failures. exists to convert as much of this as possible into scheduled maintenance by predicting faults before they land.

This is the cost side of the scale-upscale-up vs scale-outScale-up makes one machine bigger (NVLink); scale-out adds more machines (InfiniBand/Ethernet). bargain, and it deserves to be stated as plainly as the benefit. Parts I and II spend sixteen chapters explaining why a tightly coupled 72-GPU domain is worth building. The same tight coupling means a fault anywhere in it interrupts everything in it. A loosely coupled cluster of eight-GPU servers loses 8 GPUs and one replica; this rack loses the domain.

Which is why the next chapter's placement decisions are also reliability decisions. A job packed into few racks gets fast collectives and a correspondingly concentrated exposure to any one rack's faults.

The defence against losing work is to save it - write the model out to storage every so often, so a crash costs only the time since the last save. But saving is not free: while you are writing, you are not training. Save constantly and the saving is all you do. Save rarely and each crash throws away hours.

In between there is a best answer, and it has a pleasing shape: as failures get more frequent, the right interval shrinks - but only as the square root, so a machine that fails four times as often needs to checkpoint only twice as often.

Write a checkpointCheckpointA periodic dump of model and optimiser state, so a failure costs the time since the last one rather than the whole run. every T seconds at a cost of C seconds each, with a mean time between failures of M. Checkpointing overhead is C/T of the run. Lost work averages half an interval per failure, so T/(2M). Add them, differentiate, and the minimum sits at T = √(2CM) - Young's 1974 first-order result, which Daly later refined to include restart cost.^{Daly, Future Generation Computer Systems}

Put the Llama 3 numbers in. With an MTBF of ~3.1 hMean time between unexpected failures Derived: 54 days over 419 unexpected interruptions. Meta does not state this figure directly; it follows from two numbers that are stated. Meta AI and a five-minute checkpoint, the optimum is about 43 minutes. That is a concrete, defensible answer to a question usually answered with "often", and it comes out of two measured inputs.

The reason the curve matters more than the number is its flatness. Near the optimum the total overhead barely changes, so being off by a factor of two costs very little - which is why "checkpoint hourly" is defensible in practice for a wide range of clusters. What the curve punishes is being off by a factor of ten, in either direction, and both mistakes are common: over-frequent checkpointing on a reliable cluster, and hour-long intervals on one that fails every twenty minutes.

Run the figure at those inputs, though, and it says the overhead at the optimum is about 23% - which cannot be squared with Meta's >90 %Effective training time Meta’s own framing, and the right one: with something breaking every few hours, the interesting measure is not uptime but the share of elapsed time that produced work nobody had to redo. Meta AI effective training time on the same MTBF. The model is not wrong; the five-minute checkpoint is. Minimum overhead is √(2C/M), so holding it under 10% at a ~3.1 hMean time between unexpected failures Derived: 54 days over 419 unexpected interruptions. Meta does not state this figure directly; it follows from two numbers that are stated. Meta AI MTBF requires a checkpoint costing about a minute, not five.

That is the whole case for asynchronous checkpointing, arrived at from the arithmetic rather than asserted: stage the write to host memory and flush it to storage in the background, and C becomes the copy rather than the persist. The square root turns a five-fold reduction in checkpoint cost into a two-fold reduction in overhead, which is a smaller return than it sounds and still the difference between a viable run and an unviable one.

A crash is loud. It is detected, the job restarts from a checkpoint, and the cost is bounded by arithmetic you just saw. The expensive failure is the one that keeps running: a stragglerStragglerOne rank that is slower than the rest, which at a collective barrier makes everyone else that slow too., one rank slower than the rest.

Because a synchronous collective cannot complete until its slowest participant arrives, one GPU running at 90% makes the entire job 10% slower - whether the job has a hundred GPUs or sixteen thousand. There is no dilution. A thermally throttling GPU, a link that has negotiated down a lane, a host with a noisy neighbour: each taxes every step of the run and none of them trips an alarm.

This is why goodputGoodputThe share of elapsed time that produced work nobody had to redo. is the metric Meta reports rather than availability. Availability asks whether the machine was up; goodput asks what fraction of elapsed time produced work nobody had to redo. Time spent recomputing after a restart, and time spent waiting at a barrier for a straggler, are both periods when every GPU was allocated, powered and busy, and the run advanced by nothing.

It also explains an operational instinct that looks wasteful from outside: eject a suspected straggler rather than diagnose it in place. Losing one rank's contribution costs a rebalance. Keeping it costs a percentage of everything, indefinitely.