Addressing and Routing

Somewhere in the cluster a piece of software called the subnet manager is running. When the fabric powers up it goes looking: it finds every switch, every adapter and every cable between them, and builds a complete map.

Then it hands out addresses and tells every switch exactly where to send traffic for every possible destination. A switch here does not work anything out for itself - it is given a table and follows it.

The advantage is that somebody knows the whole shape of the network and can pick sensible routes through it. The cost is that this somebody now matters a great deal: while it is gone, traffic already flowing carries on, but nothing new can be worked out - a swapped cable or a replaced switch has nobody to tell the fabric about it.

The subnet managersubnet managerOne centralised service that discovers every device in an InfiniBand fabric, assigns addresses and computes every forwarding table. is "a centralized entity running on the server that discovers and configures all the InfiniBand fabric devices to enable traffic flow throughout the fabric".^{NVIDIA Docs} It assigns each port a LIDLIDLocal Identifier - the address a subnet manager assigns to a port, and what switches actually forward on. - a subnet-local address, allocated rather than burned in, distinct from the permanent GUID - and computes a forwarding table for every switch, indexed by destination LID.

So a route is a table lookup, not a computation. There is no learning, no flooding, no spanning tree and no convergence delay after a topology change, because nothing is discovered locally. The trade is that a topology change means the manager recomputes and redistributes tables, and the time that takes scales with the fabric.

Centralised routing is viable here for a reason that does not generalise: an InfiniBand subnet is administratively singular. One owner, one physical site, a topology that is known in advance because somebody drew it, and a device count in the thousands rather than the billions. None of that is true of the internet, which is why the internet distributes its control plane and pays convergence costs instead.

What it buys is worth the constraint. Because the manager knows the whole graph, it can compute routes that are deterministic and balanced by construction - and it can hand a topology-aware collectives library the actual topology rather than a guess. That is the precondition for both rail alignment and in-network reduction: neither works if nobody knows where anything is.

A fixed table has one weakness. If it always sends the same kind of traffic the same way, then when a lot of that traffic appears at once it all queues behind itself - while an equally good parallel route sits empty.

Adaptive routing lets a switch break the rule when it sees that happening: same destination, different next hop, chosen because the prescribed one is busy.

Adaptive routingadaptive routingLetting a switch send a packet down a different equal-cost port than the routing table prescribes, when the prescribed one is congested. allows a switch to forward down any port that still reaches the destination, rather than only the one the table names. In a fat tree every upward port reaches everything, so the freedom is real and the choice can be made on occupancy.

The cost is reordering. Packets of one flow now take different paths and can arrive out of order, which the transport has to tolerate - one of several places where RDMA's tight coupling to the fabric shows. On NVIDIA fabrics adaptive routing is part of the core subnet manager implementation rather than an add-on: since UFM v6.4 it no longer needs a separate plugin, and neither does SHIELDSHIELDSelf-healing: switches recognise a failed link and route around it without waiting for the subnet manager to recompute..^{NVIDIA Docs}

Adaptive routing is not a nicety on a lossless fabric - it is load-bearing. On a dropping network a congested link degrades locally: the queue fills, packets are discarded, senders back off. On a lossless one the queue fills and backpressure propagates, so a single hotspot can stall senders that were never using the congested link at all. Moving flows off the hotspot is the only mechanism that stops that spread rather than merely bounding it, which is why the previous chapter ends by pointing here.

SHIELD addresses the other failure mode. Recomputing fabric-wide forwarding tables takes time proportional to the fabric, which is far too slow to absorb a cable failure in the middle of a training run. SHIELD lets neighbouring switches agree a local detour immediately, so a link failure degrades a job rather than killing it, and the manager recomputes at its leisure.^{NVIDIA Docs}