Chapter 05

NVLink & NVSwitch

This is the chapter the rest of the rack exists to serve. The arithmetic is small enough to do in your head, and once you have done it the design stops looking like a specification and starts looking like the only sensible answer.

Eighteen and eighteen

Every GPU has eighteen separate connections leaving it. The rack contains eighteen switch chips. Wire the first connection of every GPU to the first switch, the second to the second, and so on, and something neat happens: every GPU is one hop from every other GPU, and no two conversations have to share a wire.

That is what people mean when they say the rack has no internal bottleneck. It is not a claim about being fast; it is a claim about there being nowhere for traffic to pile up.

Each Blackwell GPU exposes 18NVLink ports per GPU NVIDIA NVLink 5 ports at 100 GB/s bidirectional, totalling 1.8 TB/sNVLink 5 bandwidth per GPU Bidirectional: 18 links × 100 GB/s. About 14× PCIe Gen5. NVIDIA . Each NVSwitch tray carries two NVSwitch5 ASICs, and with 9NVSwitch trays ServeTheHome trays that is 18NVSwitch5 ASICs 2 per switch tray. NVIDIA ASICs — exactly the port count per GPU.^NVIDIA

Connect port n of each GPU to switch n, and every switch sees all 72 GPUs exactly once. Each ASIC has 36 + 36 ports and 28.8 Tb/sBandwidth per NVSwitch5 ASIC 7.2 TB/s; 36 + 36 ports. Lower than a raw 51.2 Tb/s switch because silicon budget goes to SHARP in-network reduction engines. NVIDIA of switching capacity. The fabric is one layer deep, fully connected, and non-blocking: any permutation of senders and receivers runs at line rate. Aggregate in-rack bandwidth is 130 TB/sAll-to-all NVLink bandwidth in-rack NVIDIA .

Selected: —
Links from this GPU: 18
Bandwidth, all ports: 1.8 TB/s
Reachable in one hop: 71 GPUs

72 GPUs × 18 ports = 1,296 endpoint links, before the spine's cable count.

This figure is interactive and needs JavaScript. The prose around it states every number it shows.

Each of a GPU's 18 NVLink ports terminates on a different NVSwitch ASIC, so all 18 are usable at once toward any set of peers. Drawn as 18 trays of four GPUs against 9 switch trays of two ASICs; link lines are schematic, the copper spine carries over 5,000 of them.

Worth stating what this removes rather than what it adds. There is no rail assignment, no leaf/spine oversubscription ratio, no bisection figure to derate collectives against, and no distinction between "near" and "far" peers within the rack. A parallelism strategy can be chosen on numerical and load-balance grounds instead of being contorted to keep high-traffic dimensions inside an 8-GPU node.

The one thing it does not remove is the rack boundary. Inside, per-GPU egress is 900 GB/s; across racks it is 50 GB/s on ConnectX-7 or 100 GB/s on ConnectX-8 — roughly a factor of nine to eighteen. Every serious deployment decision at this scale is about which model dimension gets to cross that line. Chapter 7 takes that up.

The switch does arithmetic

An 28.8Bandwidth per NVSwitch5 ASIC 7.2 TB/s; 36 + 36 ports. Lower than a raw 51.2 Tb/s switch because silicon budget goes to SHARP in-network reduction engines. NVIDIA Tb/s switch ASIC looks underspecified next to a 51.2 Tb/s Ethernet switch of the same era. The missing capacity went somewhere deliberate.

NVSwitch carries SHARP engines, which perform reductions and multicast inside the switch. In an all-reduce — the collective that dominates distributed training — partial results are combined as they pass through the fabric instead of being shuttled between GPUs and summed at the endpoints. That roughly halves the bytes on the wire and removes a round trip from the critical path.

Trading port bandwidth for in-network compute is the right trade when your dominant traffic pattern is a collective rather than point-to-point. It is also why the spine's quoted AllReduce figure, 260 TB/sAllReduce bandwidth over the spine NVIDIA Technical Blog , is higher than the all-to-all figure of 130 TB/sAll-to-all NVLink bandwidth in-rack NVIDIA : they are measuring different operations, not disagreeing.

What the bandwidth is worth

Bandwidth numbers are hard to feel. This converts them into the thing you actually wait for: the time to complete one all-to-all exchange, the pattern that a mixture-of-experts layer performs on every forward pass.

Payload per GPU: 256 MB
Egress bandwidth: 900 GB/s
Exchange time: 0.28 ms
Versus NVLink: 1.0×

Payload per GPU256 MB

This figure is interactive and needs JavaScript. The prose around it states every number it shows.

A first-order model: time ≈ bytes each GPU must send ÷ that GPU's egress bandwidth. NVLink's 1.8 TB/s is a bidirectional figure, so egress is 900 GB/s; the scale-out numbers are per-SuperNIC line rates. Real collectives add protocol and synchronisation overhead, so treat these as lower bounds and as ratios rather than as predictions.

Interconnect figures

NVLink 5 and NVSwitch5
NVLink 5 bandwidth per GPU	1.8 TB/s	Bidirectional: 18 links × 100 GB/s. About 14× PCIe Gen5.
NVLink ports per GPU	18
Bandwidth per NVSwitch5 ASIC	28.8 Tb/s	7.2 TB/s; 36 + 36 ports. Lower than a raw 51.2 Tb/s switch because silicon budget goes to SHARP in-network reduction engines.
NVSwitch5 ASICs	18	2 per switch tray.
All-to-all NVLink bandwidth in-rack	130 TB/s
AllReduce bandwidth over the spine	260 TB/s
Copper cables in the NVLink spine	>5,000	± sources disagree: 5,000 – 5,184. NVIDIA’s OCP contribution says "over 5,000"; other sources cite 5,184. Jensen Huang described it as "5,000 NVLink cables. In total, 2 miles."
NVLink cable cartridges	4
Power an all-optical spine would have cost	~20 kW	NVIDIA’s stated rationale for copper. SemiAnalysis independently computed ~19.4 kW from 648 × 1.6T transceivers at ~30 W.

Copper, and the 20 kW that decided it

The spine could have been optical. It is not, and the reason is a power calculation: transceivers and retimers for the NVLink spine would have drawn roughly ~20 kWPower an all-optical spine would have cost NVIDIA’s stated rationale for copper. SemiAnalysis independently computed ~19.4 kW from 648 × 1.6T transceivers at ~30 W. SemiAnalysis .^SemiAnalysis SemiAnalysis reached ~19.4 kW independently, counting 648 1.6T transceivers at about 30 W each.

Against ~120 kWRack powerSources disagree: 120 kW nominal · 125–135 kW operating (Supermicro) · 132 kW fully loaded (Schneider Electric) Supermicro’s datasheet states an operating power of 125–135 kW and 132 kW of installed power-shelf capacity. Steven Carlini, writing for Schneider Electric: "When fully loaded into a rack, the latest NVIDIA-based GPU servers require 132 kW of power." The commonly quoted ~120 kW is the nominal design figure, not a measured ceiling. Supermicro · Steven Carlini, Schneider Electric — Forbes Technology Council · ServeTheHome that is about a sixth of the rack's budget spent moving data a metre and a half. At that reach copper is both far cheaper and far more efficient — and the price of using it is that the switches must sit in the middle of the rack and the trays must be close-packed around them. The rack's physical arrangement is downstream of this one decision.