Collectives on the Wire

Training works like this: every GPU computes a suggested adjustment to the model, and then all of them have to agree on the sum of those suggestions before anyone can continue. That agreement is called an all-reduce, and it happens constantly.

The obvious way to do it is to pass partial totals around a circle until everyone has seen everything, which means the same numbers cross the network many times. The clever way is to let the network add them up on the way past - so each GPU sends its contribution once, and gets the answer back.

An all-reduce combines one value per participant and delivers the result to all of them. Its cost is the reason interconnect matters at all in training, and there are three ways to pay it.

A ring passes partial sums around a cycle: bandwidth-optimal in total bytes but with a step count proportional to the number of participants, so latency grows linearly. A tree reduces up and broadcasts down in logarithmic steps - far better latency, at the cost of more total traffic. Both do the arithmetic at the endpoints, which means every intermediate result travels the fabric.

SHARPSHARPScalable Hierarchical Aggregation and Reduction Protocol - the switch does the arithmetic of a collective in-network. does it in the switch. Partial results are combined inside the ASIC as they pass through, so an endpoint sends its contribution once and receives the answer once, and the intermediate values never appear on the wire at all.

The point worth carrying is that this is a traffic elimination mechanism, not a transport optimisation. Ring and tree both move the same data repeatedly and differ only in how they schedule it; in-network reduction changes how many bytes exist. That is why it composes with everything else rather than competing with it.

It composes with itself, too. Chapter 5 of Part I describes SHARP engines inside the in-rack NVSwitch; Quantum-X800 carries v4SHARP version on Quantum-X800 In-network reduction on the scale-out switch. Distinct from, and composable with, the SHARP engines inside the in-rack NVSwitch. NVIDIA on the scale-out switch.^NVIDIA A cluster-wide all-reduce can therefore reduce within each rack over NVLink first and then across racks over InfiniBand - two tiers of in-network reduction, so what crosses the expensive tier is one value per rack rather than one per GPU.

In-network reduction is not free silicon. Part I makes the point about NVSwitch: an 28.8 Tb/sBandwidth per NVSwitch5 ASIC 7.2 TB/s; 36 + 36 ports. Lower than a raw 51.2 Tb/s switch because silicon budget goes to SHARP in-network reduction engines. NVIDIA ASIC looks underspecified beside a raw 51.2 Tb/s switch of the same era, and the difference is silicon that could have been ports and went to reduction engines instead.

The same trade is made on the scale-out switch. You are buying arithmetic units, buffering and the control logic to build aggregation trees, and paying for them in port count or die area. It is a good trade exactly when collectives dominate your traffic - which is the defining property of this workload and not of most others.

There is an operational dependency here worth naming. An aggregation tree has to be constructed over the physical topology before it can be used, which requires something that knows the topology - the subnet manager. In-network reduction is therefore downstream of centralised routing: it is one of the capabilities that a single authority over the fabric buys, and it is hard to offer on a fabric where nobody has the whole map.

NCCLNCCLNVIDIA’s collective communications library - the thing that actually executes an all-reduce, and it is topology-aware. is what actually selects among ring, tree and SHARP, from the topology it discovers. Almost nothing above it makes that choice explicitly, which is why fabric decisions surface to model engineers as throughput rather than as code - and why a fabric that reports its topology accurately is worth more than one that merely runs fast.