InfiniBand versus Spectrum-X Ethernet

Start with what does not distinguish them, because it is more than people expect.

The programming model is identical. RoCERoCERDMA over Converged Ethernet - the same verbs and queue pairs, carried by Ethernet instead of InfiniBand. carries the same RDMA semantics over Ethernet: the same queue pairsqueue pairA send queue and a receive queue, together forming one RDMA connection endpoint., the same verbsverbsThe de-facto RDMA programming interface: post a work request, poll a completion, never enter the kernel on the data path., the same registered regions. GPUDirect RDMA works over both, so a GPU buffer is a valid target either way.^{NVIDIA CUDA Docs} Nothing above the adapter is written differently.

Line rates match too: 800 Gb/sXDR per-port rate Four lanes at 200 Gb/s. Quantum-X800 generation. NVIDIA · NVIDIA per port on Quantum-X800, and 800 G ports on Spectrum switches, with the same ConnectX-8 SuperNIC available on both sides.^{NVIDIA,NVIDIA} Radix is comparable and in one case better - a 512 × 800Spectrum SN6800 Ethernet switch In a 5U chassis - the highest radix on either side of the InfiniBand/Ethernet line. NVIDIA Gb/s SN6800 has the highest port count on either side of the line.

That the programming model is shared is the fact that reframes the whole comparison. If RDMA, GPUDirect and NCCL behave the same on both, then this is not a decision about software portability and never was. It is a decision about the properties of the wire underneath identical software - and about operations.

Both networks refuse to drop packets, because the workload cannot tolerate it. They get there differently.

InfiniBand was built this way from the start: a sender is told how much room there is and never exceeds it. Ethernet was built to drop, so losslessness has to be added - by having a congested switch shout "stop" backwards, and by marking traffic so senders slow down before it comes to that.

Both end up refusing to drop packets. The difference is that the Ethernet version is bolted on, with settings that have to be right - and when they are not, the shouting spreads. A switch told to stop tells the switches behind it to stop, and traffic that had nothing to do with the original congestion stalls too.

The difference is proactive versus reactive. Credits are permission granted in advance, so overflow is structurally impossible.^{Mellanox Technologies, presented to IEEE 802.1} PFCPFCPriority Flow Control - Ethernet’s pause frame, per traffic class: "stop sending" rather than "here is how much you may send". is a pause sent once buffers are already filling, with ECNECNExplicit Congestion Notification - marking packets rather than dropping them, so the sender slows before anything is lost. marking packets end-to-end so senders reduce their rate before it comes to that. Both deliver a lossless fabric. PFC is the one with thresholds to tune and documented ways to fail.

Those failures are worth naming rather than gesturing at, because they are the reason lossless Ethernet has a reputation. A PFC pause travels backwards hop by hop, so a single congested link can pause switches carrying flows that never touched it - congestion spreads to victims. Cyclic buffer dependencies between paused ports can deadlock a region of the fabric outright. And a single misbehaving adapter emitting pause frames can quiet its whole neighbourhood. Microsoft's production account of running RoCEv2 at scale reports all three: "the safety challenges brought by PFC-induced deadlock (yes, it happened!), RDMA transport livelock, and the NIC PFC pause frame storm problem".^{Guo et al., Microsoft - ACM SIGCOMM}

None of these is a reason not to use Ethernet - the same paper is a report of solving them, and Spectrum-X exists partly because they are solvable. It is a reason the two options are not equivalent work: credits make the failure structurally impossible, while PFC makes it a thing you configure against.

The second difference is who is in charge. InfiniBand has a subnet managersubnet managerOne centralised service that discovers every device in an InfiniBand fabric, assigns addresses and computes every forwarding table. with a complete map of the fabric. Spectrum-X gets its equivalent behaviour from the switch and SuperNIC working "in tight coordination" to route adaptively.^NVIDIA Centralised knowledge versus coordinated endpoints - a real architectural difference, with real consequences for both what the fabric can offer and what has to be running for it to work.

The consequences of that second difference are the ones worth planning around. A central authority that knows the whole topology is what makes deterministic routing, topology-aware collective selection and in-network aggregation trees straightforward to offer. It is also a component whose availability matters, and a recomputation cost that scales with the fabric.

Ethernet brings the opposite set. No single authority to depend on, an enormous operational ecosystem, staff who already know it, and multi-tenancy that InfiniBand addresses less naturally - NVIDIA positions Spectrum-X explicitly for "multi-tenant generative AI factories".^NVIDIA For a single-tenant training cluster that is not worth much. For a cloud provider selling slices of one, it may be the whole argument.