Fat-Trees, Rails and Oversubscription

Switches have a fixed number of sockets. A Q3400 has 144. If you need to connect more machines than that, you have to connect switches to each other - and now some of those 144 sockets are used up talking to other switches instead of to machines.

Arrange them in a tree and a problem appears: everything at the bottom eventually has to squeeze through the top. A fat tree fixes that by making the upper branches as thick as everything below them, so nothing narrows on the way up. That is more switches and far more cable than a thin tree - and it is the difference between a network that slows down when everyone talks at once and one that does not.

The arithmetic is simple enough to do in your head. Take a switch of radix k and point half its ports down at endpoints and half up at the tier above. A two-tier fat tree then holds k × k/2 endpoints: at k = 144 that is 144 × 72 = 10,368, which is why NVIDIA quotes over 10,000Endpoints in a two-tier fat tree NVIDIA’s figure for Quantum-X800. The arithmetic behind it: a radix-144 switch with half its ports facing downward gives 144 × 72 = 10,368 endpoints before a third tier is needed. NVIDIA for a two-level fat tree on Quantum-X800.^NVIDIA

The property that matters is uniformity. Every endpoint is the same number of hops from every other, and every pair has a path that does not contend with every other pair. For a collective - whose completion time is set by its slowest participant - a uniform fabric is worth more than a fast but lumpy one.

Radix is the lever with the most leverage, because tiers multiply cost non-linearly. Each additional tier adds two hops to every cross-fabric path, adds a full set of switches, and - since inter-tier links cross rows - turns copper into optics. Doubling radix from 72 to 144 quadruples two-tier capacity, which is why switch vendors chase port count rather than per-port rate alone.

It is also why NVIDIA's fabrics are not only fat trees. The reference architecture notes support for "fat-tree, Dragonfly, and multi-dimensional Torus" topologies.^{NVIDIA Docs} Fat trees dominate for AI because they are uniform and because collectives are the workload; the alternatives trade uniformity for cable count once a fat tree would need a third tier to fit the endpoints.

OversubscriptionoversubscriptionDeliberately fitting less upward bandwidth than downward, on the bet that not everyone talks upward at once. means fitting less upward capacity than downward - pointing more of each leaf's ports at endpoints and fewer at the tier above. A 2:1 upper tier roughly halves the spine switches and the optics between tiers. Put the builder above at radix 144, two tiers, 1:1 and it reports 72 spine switches and 10,368 inter-tier cables; at 2:1 that becomes 48 and 6,912. Those saved cables are the ones crossing rows, so they are the expensive optical ones rather than the copper.

It is invisible to traffic that stays local, which is most enterprise traffic and almost none of this. An all-reduce is the pattern in which every endpoint transmits across the fabric simultaneously - precisely the case oversubscription bets against. Which is why AI fabrics are usually built non-blocking where general-purpose data-centre fabrics are not, and why the SuperPOD design is explicit about being "a fully non-blocking fat tree topology for each SU".^{NVIDIA Docs}

Bisection bandwidthbisection bandwidthThe bandwidth crossing the worst-case cut that splits the fabric in half - the honest measure of an all-to-all. is the honest measure and aggregate bandwidth is the flattering one. Aggregate counts every link in the fabric, including the ones a given traffic pattern will never use; bisection counts only what crosses the worst-case cut, which is exactly what an all-to-all has to push through.

Note what adaptive routing can and cannot do about it. It chooses among paths that exist, so on an oversubscribed fabric it will distribute contention evenly across uplinks that are collectively too few. Even distribution of an inadequate resource is still inadequate. Oversubscription is a capacity decision, not a scheduling one, and no amount of cleverness at run time recovers it.

Here is a trick that costs nothing in hardware and removes two hops from the traffic that matters most. Each compute tray has four network connections, one per GPU. Instead of wiring them arbitrarily, wire GPU number one in every rack to the same switch, GPU number two in every rack to a second switch, and so on.

Now when the GPUs numbered one across the whole cluster want to talk to each other - which is exactly what they do during a collective - they are all already plugged into the same switch. The traffic never climbs the tree at all.

That is rail alignmentrail-optimisedWiring the same-numbered GPU in every rack to the same leaf switch, so a collective along that rail crosses one switch instead of climbing the tree.. A compute tray carries 4InfiniBand rails per compute tray One per GPU - the tray carries four ConnectX-7 NICs. Same-numbered GPUs across racks share a rail, so "traffic per rail of each compute tray is always one hop away from other compute trays in the same Scalable Unit". NVIDIA Docs · NVIDIA Docs - one per GPU - and racks are rail-aligned so that, in NVIDIA's words, "traffic per rail of each compute tray is always one hop away from other compute trays in the same Scalable Unit".^{NVIDIA Docs}

The structure follows from it. Each Scalable Unit contains four spine-leaf groups, one per rail, and each group has 8Leaf switches per spine-leaf group One for each compute rack. NVIDIA Docs - one per compute rack - and 6Spine switches per spine-leaf group Giving a fully non-blocking fat tree per Scalable Unit, attached to six core groups. NVIDIA Docs .^{NVIDIA Docs} The rail count and the group count are the same number for the same reason.

This is the scale-out restatement of the trick that makes the in-rack fabric single-layer. Chapter 5 of Part I matches NVLink port n on every GPU to NVSwitch n, so any peer is one hop away. Rail alignment matches NIC n on every tray to leaf switch n, so any same-indexed peer is one hop away. Same reasoning, one tier out: collectives dominate, so build the topology the collective wants rather than a general-purpose one.

It also pays for itself physically. A rail hop is tray-to-leaf, which the layout can usually keep inside the copper envelope; a hop across tiers crosses rows and must go optical. Rail alignment converts the dominant traffic of a collective into the cheapest link in the building.