What It Costs

A rack is an appliance that converts electricity into text. So the cost of the text is the cost of the electricity, plus a share of the cost of the appliance, divided by how much text you got. Three numbers.

Two of them are easy to find. Electricity has a price and the rack has a power draw. The third - how much text - is the one nobody publishes.

Energy per token is power divided by throughput. At ~120 kWRack powerSources disagree: 120 kW nominal · 125–135 kW operating (Supermicro) · 132 kW fully loaded (Schneider Electric) Supermicro’s datasheet states an operating power of 125–135 kW and 132 kW of installed power-shelf capacity. Steven Carlini, writing for Schneider Electric: "When fully loaded into a rack, the latest NVIDIA-based GPU servers require 132 kW of power." The commonly quoted ~120 kW is the nominal design figure, not a measured ceiling. Supermicro · Steven Carlini, Schneider Electric - Forbes Technology Council · ServeTheHome and a hypothetical 10,000 tokens per second, that is 12 joules per token, or about 3.3 kWh per million tokens. Multiply by an electricity price and you have the energy component; divide the rack's capital cost over its depreciation period and its expected utilisation, and you have the rest.

Two corrections make it honest. Facility overhead - cooling, distribution, losses - adds the PUEPUEPower Usage Effectiveness - total facility power divided by IT power. 1.0 is the unreachable ideal. multiplier from chapter 6 of Part I. And a fleet does not run flat out, for exactly the reason the previous chapter gives: the headroom that keeps latency inside its target also spreads the fixed cost over fewer tokens.

Notice which term dominates. At any plausible electricity price the capital cost of the hardware, amortised, is several times the energy - which is why utilisation matters more to cost per token than efficiency does, and why the scheduling chapter's fragmentation is an economic problem rather than an aesthetic one. An idle GPU still depreciates.

It is also why 38–43 %Model FLOPs Utilisation, Llama 3 405B BF16 MFU over the whole run. The single most useful number on this site for calibrating expectations: after every architectural decision Parts I and II describe, a real training job converts under half of peak FLOPS into model arithmetic. Meta AI MFU and a 60%-utilised serving fleet compound into something worse than either: the denominator of the cost calculation is not the machine's peak but the product of every fraction in this part.

The throughput slider has no default worth trusting, because no absolute tokens-per-second figure for GB200 NVL72 appears in vendor material. What appears instead is ratios. NVIDIA's inference claims are 30×Real-time trillion-parameter LLM inference NVIDIA’s measured configuration: TTL = 50 ms, FTL = 5 s, 32,768 input / 1,024 output tokens, GPT-MoE-1.8T, comparing 64 Hopper GPUs over InfiniBand against 32 Blackwell GPUs in an NVL72. Not a like-for-like GPU count. NVIDIA · NVIDIA Technical Blog and 10×Mixture-of-experts performance Same measured configuration as the 30× figure. NVIDIA , both against a named previous generation. The disaggregation gain is ~1.5× throughputDisaggregated vs aggregated serving MLPerf Inference v5.1, Llama 3.1 405B interactive. MLCommons · NVIDIA · NVIDIA Technical Blog . The GB300 claim is 2.7×GB300 NVL72 token throughput uplift NVIDIA’s figure, from the same infrastructure and power footprint, with a claimed >60% reduction in cost per token. A ratio with no published denominator: no absolute tokens-per-second and no currency figure accompanies it. NVIDIA - "2.7x more tokens from the same infrastructure and power footprint, reducing the cost to manufacture each token by more than 60%".^NVIDIA

Every one of those is a numerator over an unstated denominator. More tokens than what, in what units, at what latency target? Cheaper by 60% than what starting price? A ratio cannot be turned into a cost, and no amount of arranging them produces an absolute.

This is not an oversight, and it is not unique to one vendor. A published absolute would be a commitment: it would fix a configuration, a precision, a request mix and a latency target, and it would be quoted back at the vendor by every customer whose workload differs. A ratio against your own previous product is defensible, unfalsifiable in practice, and markets just as well.

Absolutes do exist, under exactly the conditions that make them costly to publish casually: MLPerf Inference submissions are measured against a fixed scenario and audited, which is why this site can quote 27.3 minLlama 3.1 405B training time MLPerf Training v5.0, 4 June 2025: 27.33 minutes on 2,496 Blackwell GPUs across 39 racks running 64 active GPUs each - not fully populated 72-GPU racks, which is why 2,496 does not divide by 72. CoreWeave puts an equivalent H100 setup at around 156 racks, assuming 32 GPUs per rack. A later round reached about 10 minutes on more than 5,000 Blackwell GPUs. CoreWeave · MLCommons for training but cannot complete the calculation above. The results database is the place to go, and reading a figure out of it means accepting its scenario as your own.

A ratio with no denominator is useless for budgeting and genuinely informative about direction. "2.7× the tokens from the same power footprint" says the thing this whole site has been arguing: at this scale the binding constraint is not capital but electricity and floor space, so the figure of merit is tokens per watt rather than tokens per dollar.

That reframing is why power and cooling is a chapter about performance rather than facilities, and why the roadmap is really a power curve. A datacentre with a fixed megawatt allocation cannot buy its way past it; the only lever left is how many tokens each watt produces.

Which closes the loop on Part I's opening claim. "One giant GPU" is not a statement about programming convenience. It is an efficiency argument: a coherent domain lets the chatty parallelism dimensions run over links that cost a fraction of the energy per bit that the scale-out fabric does, and it raises the MFU those watts are multiplied by. The rack exists to move the tokens-per-watt number, and every chapter of this site has been describing one term in it.