Glossary · NVL72

Terms are defined here once and hover-linked everywhere else. Definitions are written to be useful rather than complete - where a term hides a real design decision, the second paragraph says what it is.

Activation recomputation

Throwing away intermediate activations during the forward pass and recomputing them during the backward pass.

Also called gradient checkpointing, and unrelated to failure-recovery checkpoints. It trades arithmetic for memory - typically about 30% more compute for a large reduction in peak memory. Because MFU counts only the FLOPs the model requires, recomputation shows up as lower MFU even though it is often the decision that makes the run possible at all.

adaptive routing

Letting a switch send a packet down a different equal-cost port than the routing table prescribes, when the prescribed one is congested.

Static routing in a fat tree is deterministic and therefore predictably bad: several flows hashed onto the same uplink collide while a parallel uplink sits idle. Adaptive routing spreads them at the cost of reordering, which RDMA transports must then tolerate. On NVIDIA fabrics it is part of the core subnet manager rather than an add-on.

bisection bandwidth

The bandwidth crossing the worst-case cut that splits the fabric in half - the honest measure of an all-to-all.

Aggregate bandwidth counts every link and flatters a badly shaped network. Bisection counts only what crosses the narrowest division, which is what an all-to-all or an expert exchange actually has to push through.

busbar

A shared conductor running the height of the rack that feeds every tray, replacing per-server power supplies.

Rack-level power shelves rectify facility input once, and trays draw from the busbar. Consolidating conversion is more efficient and reclaims the volume that 18 pairs of redundant PSUs would occupy.

CDU

Coolant Distribution Unit - the pump-and-heat-exchanger that isolates the rack’s coolant loop from the facility’s.

It can sit in-rack (typically at the bottom) or in a sidecar. It gives the rack a controlled, filtered loop at a known pressure and temperature independent of whatever the building water is doing.

Checkpoint

A periodic dump of model and optimiser state, so a failure costs the time since the last one rather than the whole run.

The interval is a genuine optimisation: checkpoint too often and the writes themselves consume the run, too rarely and each failure discards more work. Young’s first-order result puts the optimum near the square root of twice the checkpoint cost times the mean time between failures.

Context parallelism

The sequence itself is split across GPUs, so one very long context can be processed by several at once.

The fourth dimension in Llama 3’s [TP, CP, PP, DP] split. Attention needs every position to see every earlier position, so splitting the sequence means exchanging keys and values between the GPUs holding neighbouring chunks. It exists because context lengths grew faster than per-GPU memory did.

Continuous batching

Admitting and retiring requests between decode steps rather than running a batch to completion.

Introduced as iteration-level scheduling in Orca. Under request-level batching a finished request cannot return until the slowest member of its batch finishes, and a newly arrived one waits for the whole batch; scheduling per iteration lets each leave as soon as it emits its stop token, and lets new work join the very next step.

credit-based flow control

The receiver tells the sender how much buffer it has; the sender may not exceed it. Congestion becomes a stall, never a drop.

Credits are carried in-band rather than as separate pause frames, and they are absolute - a running total of what has been allowed since the link came up - rather than incremental, so a lost credit update self-corrects on the next one instead of desynchronising the link forever. That resiliency is the hard part of the design, not the accounting.

See also lossless fabric, virtual lane, PFC

Data parallelism

Every GPU holds the whole model and a different slice of the batch; gradients are all-reduced each step.

The oldest and simplest split, and the one that scales furthest, because its only communication is one all-reduce of the gradients per step. Its cost is memory: a full copy of parameters, gradients and optimiser state per replica, which is what sharded optimisers exist to avoid.

decode

The token-by-token generation phase. Memory-bandwidth-bound, not compute-bound.

Decode generates one token at a time, so each step reads the entire weight set (and the KV cache) to do very little arithmetic. Throughput is set by memory bandwidth, which is why batching many requests together matters so much here and not in prefill.

disaggregated serving

Running prefill and decode on different GPUs, each tuned for its own bottleneck.

NVIDIA Dynamo splits the two phases across separate GPU pools and streams the KV cache between them. Because they have opposite bottlenecks, co-locating them forces a compromise; separating them let GB200 NVL72 reach roughly 1.5× the throughput of aggregated serving on MLPerf Inference v5.1.

DPU

A data processing unit - BlueField-3 here - carrying storage, management and isolation on its own network, away from the compute fabric.

Worth separating from the SuperNIC in your head: the two BlueField-3 adapters in a compute tray are not part of the GPU fabric at all. They serve in-band management and storage, so the compute fabric can be dedicated entirely to collectives.

ECN

Explicit Congestion Notification - marking packets rather than dropping them, so the sender slows before anything is lost.

ECN works end to end where PFC works hop by hop, so the two are usually deployed together: PFC to protect the buffer in the moment, ECN to reduce the offered load so PFC stops being needed.

expert parallelism

Spreading an MoE model’s experts across many GPUs so each holds only a few.

Wide expert parallelism reduces the weight memory each GPU must hold, which lets you raise batch size and thus decode throughput. It is only affordable when the all-to-all between GPUs is nearly free - the 130 TB/s domain is what makes it so.

fat tree

A tree whose upward capacity does not narrow: every tier has as much bandwidth leaving it as arriving.

Built from identical fixed-radix switches, a fat tree gives full bisection bandwidth and a uniform hop count between any two endpoints - which is exactly what a collective wants, since its completion time is set by its worst pair. Two tiers of a radix-144 switch reach past 10,000 endpoints; a third tier is what takes you to the tens of thousands.

FP4 / NVFP4

A 4-bit floating-point number format. Blackwell’s tensor cores execute it natively.

Halving the bits roughly doubles both arithmetic throughput and the number of weights that fit in memory and on the wire. NVFP4 is NVIDIA’s block-scaled variant, which keeps accuracy usable by attaching a shared scale factor to small groups of values.

Fragmentation

Free GPUs that are the wrong shape - enough capacity in total, but not enough of it adjacent.

The direct cost of insisting on locality. A cluster can sit at 85% allocated and still refuse a 72-GPU job because no single NVLink domain has 72 free. Draining and repacking recovers the shape at the price of throughput, which is why placement policy is a scheduling decision rather than a solved problem.

Gang scheduling

Starting all of a job’s ranks together or not at all, because a collective cannot proceed with some of them missing.

A tightly coupled job is not divisible: 512 GPUs now is useful, 256 now and 256 in an hour is not. Gang scheduling is what makes a scheduler hold resources idle waiting for the rest of an allocation - the behaviour that produces fragmentation.

Goodput

The share of elapsed time that produced work nobody had to redo.

Uptime counts whether the machine was running; goodput counts whether the running was useful. At scale they diverge sharply, because time spent recomputing work lost to a failure, or waiting on a straggler, is time the cluster was fully busy and produced nothing. Meta reports better than 90% effective training time for Llama 3 on this basis.

GPUDirect RDMA

The adapter reads and writes GPU memory directly, so a transfer never bounces through host memory at all.

Without it, a GPU-to-GPU transfer across the fabric stages through system RAM twice - once on each side - and the host memory bandwidth becomes the ceiling. The `nvidia-peermem` module registers GPU memory with the InfiniBand subsystem to make this possible, and it works over RoCE as well as InfiniBand.

See also RDMA, memory registration, RoCE

HBM3e

High Bandwidth Memory - DRAM stacked vertically next to the GPU die, trading capacity for enormous bandwidth.

Each Blackwell package carries 192 GB of HBM3e delivering 8 TB/s. Stacking the DRAM on an interposer millimetres from the die is what makes that bandwidth physically possible; it is also why capacity is small compared to a CPU’s DDR and why the memory is soldered rather than socketed.

head-of-line blocking

A packet that cannot move prevents the packets queued behind it from moving either, even where those could proceed.

This is the bill for never dropping. A lossy network resolves congestion by discarding the blocked packet; a lossless one holds it, and the hold propagates backwards through the fabric as credits stop being returned. Virtual lanes and adaptive routing exist to keep that backpressure from spreading further than it must.

KV cache

The stored keys and values from every previous token, re-read on each decode step.

It grows linearly with context length and with batch size, and it is read in full for every generated token - which makes it a first-order consumer of both HBM capacity and HBM bandwidth in long-context serving.

LID

Local Identifier - the address a subnet manager assigns to a port, and what switches actually forward on.

A LID is subnet-local and assigned, not burned in; the permanent per-device identifier is the GUID. Switch forwarding tables are indexed by destination LID, so a route is a table entry rather than a computed next hop.

lossless fabric

A network that never drops a packet for lack of buffer space, because a sender is not allowed to send until space exists.

Ethernet was designed to drop under congestion and let a higher layer notice and retransmit. That is a reasonable trade when the traffic is independent flows, and a terrible one when it is a collective: an all-reduce finishes when its slowest participant finishes, so one retransmission timeout stalls every GPU in the job. InfiniBand removes the possibility rather than the recovery path.

memory registration

Pinning a memory region and handing the adapter the keys to it, so it can be read or written without the OS in the loop.

This is where the real cost of RDMA sits. Registration is expensive and the region must stay resident, so high-performance stacks register large buffers once and reuse them rather than registering per message. A benchmark that registers inside its timing loop measures the wrong thing.

MFU

Model FLOPs Utilisation - the share of a machine’s peak arithmetic that ends up doing the model’s own maths.

Take the FLOPs the model definition requires for a step, divide by the FLOPs the hardware could have done in the time that step took. Everything else - communication, pipeline bubbles, recomputation, memory stalls - lands in the gap. It is deliberately unforgiving: recomputed activations are counted once, as the model needs them, not twice as the hardware did them. Meta reports 38-43% for Llama 3 405B, which is a good result, not a poor one.

MIG

Multi-Instance GPU - hardware partitioning of one physical GPU into isolated slices.

MNNVL

Multi-Node NVLink - exporting GPU memory across separate operating-system domains within the NVLink fabric.

The IMEX service brokers the export/import handshake; Kubernetes models it with a ComputeDomain custom resource via the GPU Operator and the DRA driver.

MoE

Mixture of Experts - a model where each token is routed to a small subset of many parallel sub-networks.

MoE buys capacity without proportional compute, but routing means every token must be shipped to whichever expert it selected and the result shipped back. That is an all-to-all exchange on every layer, which is exactly the traffic pattern NVLink is good at and Ethernet is bad at.

NCCL

NVIDIA’s collective communications library - the thing that actually executes an all-reduce, and it is topology-aware.

NCCL discovers the topology it is running on and picks an algorithm to match: ring or tree inside a coherent domain, and in-network reduction where the switch offers it. Almost nothing above it - a training framework, an inference server - chooses a collective algorithm directly, which is why fabric decisions show up as throughput rather than as code changes.

non-blocking

Every port can run at full rate simultaneously - no traffic pattern can starve another.

Because each of a GPU’s 18 NVLink ports lands on a different switch ASIC, and each ASIC has enough ports for all 72 GPUs, any permutation of senders and receivers is served at line rate. There is no oversubscription ratio to reason about.

NVLink

NVIDIA’s GPU-to-GPU interconnect - on NVL72 it carries 1.8 TB/s per GPU, about 14× PCIe Gen5.

NVLink 5 gives each GPU 18 ports of 100 GB/s. Crucially it is load/store coherent, not packet-and-DMA like a network: a kernel can read another GPU’s memory with an ordinary pointer dereference.

NVLink-C2C

The 900 GB/s coherent link between a Grace CPU and its Blackwell GPUs, replacing PCIe.

Coherence is the operative word: the CPU and GPU share one address space with hardware-maintained cache coherence, so there is no explicit copy step and no pinned-memory dance.

NVSwitch

The crossbar that turns point-to-point NVLink into an any-to-any fabric.

Each NVSwitch5 ASIC moves 28.8 Tb/s across 36 + 36 ports. Eighteen of them, one per GPU NVLink port, produce a single-layer non-blocking fabric across all 72 GPUs.

OCP / ORv3

The Open Compute Project and its Open Rack v3 standard - openly licensed mechanical and power specifications.

NVL72 is ORv3-inspired, and NVIDIA contributed elements of the design back to OCP. Those openly licensed documents are the legitimate source for dimensions.

oversubscription

Deliberately fitting less upward bandwidth than downward, on the bet that not everyone talks upward at once.

A 2:1 upper tier halves the switches and optics you buy and is invisible on traffic that stays local. It is not invisible to an all-reduce, which is precisely the pattern that does make everyone talk at once - which is why AI fabrics are usually built non-blocking where general-purpose data-centre fabrics are not.

PFC

Priority Flow Control - Ethernet’s pause frame, per traffic class: "stop sending" rather than "here is how much you may send".

Pause is reactive where credits are proactive: it acts only once buffers are already filling, and it travels backwards hop by hop, so congestion spreads to flows that never touched the busy link. Three failure modes are documented from production RoCEv2 deployments - deadlock from cyclic buffer dependencies, transport livelock, and pause-frame storms from a single misbehaving adapter. PFC is what makes lossless Ethernet possible, and what makes it an engineering project.

Pipeline bubble

The idle time at the start and end of a pipelined step, while stages wait for work to reach them.

With p stages and m micro-batches the bubble is about (p-1)/(m+p-1) of the step. More micro-batches shrink it, which is why pipeline parallelism and large batches travel together, and why a deep pipeline on a small batch is a bad trade.

Pipeline parallelism

Consecutive layers live on different GPUs, and micro-batches flow through them like a factory line.

Communication is small - one activation tensor per stage boundary - which makes it the natural dimension to send across a slower link. The cost is the bubble: at the start and end of every step some stages have nothing to work on.

prefill

The first phase of inference: processing the whole prompt at once. Compute-bound.

Prefill runs one big matrix multiply per layer over every prompt token in parallel, so it saturates the tensor cores. It sets time-to-first-token.

PUE

Power Usage Effectiveness - total facility power divided by IT power. 1.0 is the unreachable ideal.

Warm-water direct-to-chip cooling can push PUE toward 1.05–1.1 because a 45 °C inlet can often be met with dry coolers and no compressor at all.

queue pair

A send queue and a receive queue, together forming one RDMA connection endpoint.

Work is posted as descriptors onto a queue rather than passed through a system call, and completions are reaped from a separate completion queue. The application talks to the adapter through memory it shares with it, which is what makes kernel bypass possible.

rail-optimised

Wiring the same-numbered GPU in every rack to the same leaf switch, so a collective along that rail crosses one switch instead of climbing the tree.

Each compute tray has one NIC per GPU, and those NICs are assigned to separate rails. Because a ring or all-reduce is naturally organised by GPU index, aligning rails to indices means the dominant traffic pattern never leaves the leaf tier. It is the scale-out counterpart of matching NVLink port n to switch n inside the rack - same reasoning, one tier out.

RAS engine

Reliability, Availability, Serviceability - on-die logic that runs self-tests and predicts failures.

RDMA

Remote Direct Memory Access - one machine’s network adapter writes directly into another machine’s memory, with neither CPU involved in the transfer.

The point is not only speed but who does the work. A conventional send copies through kernel buffers and wakes the receiving process; RDMA hands the adapter a pre-registered region and a descriptor, and the adapter does the rest. At 400 or 800 Gb/s per port there is no CPU budget for anything else.

See also queue pair, verbs, GPUDirect RDMA

RoCE

RDMA over Converged Ethernet - the same verbs and queue pairs, carried by Ethernet instead of InfiniBand.

RoCE keeps the programming model and changes the wire, which means it inherits Ethernet’s willingness to drop and has to be talked out of it with PFC and ECN. That is the whole difficulty: RDMA assumes a lossless link, and Ethernet does not natively provide one.

roofline

A model that says performance is capped by either compute or memory bandwidth, whichever binds first.

Plot achievable FLOPS against arithmetic intensity (FLOPs per byte moved) and you get a slanted memory-bound ramp meeting a flat compute-bound ceiling. The corner - the "ridge point" - is the intensity a workload needs to be worth its FLOPS. On modern accelerators that ridge sits high, which is why so many real workloads live on the ramp.

scale-up vs scale-out

Scale-up makes one machine bigger (NVLink); scale-out adds more machines (InfiniBand/Ethernet).

NVL72’s whole thesis is that scale-up now extends to a whole rack. Inside the rack a GPU has 900 GB/s of egress; outside it has 100 GB/s on ConnectX-8 or 50 GB/s on ConnectX-7 - a factor of 9 or 18 depending on the generation. Partitioning a model across those two tiers is the central performance decision.

Sharded optimiser state

Splitting optimiser state, gradients and parameters across data-parallel ranks instead of replicating them.

Adam keeps two moments per parameter, so optimiser state typically dwarfs the weights themselves. Sharding it across the data-parallel group turns a per-GPU cost into a per-group one, at the price of gathering what each GPU needs when it needs it. ZeRO and FSDP are the well-known implementations.

SHARP

Scalable Hierarchical Aggregation and Reduction Protocol - the switch does the arithmetic of a collective in-network.

For an all-reduce, partial sums are combined inside the NVSwitch ASICs as data passes through, rather than being shuttled back and forth between GPUs. This halves the bytes on the wire and is why the NVSwitch is 28.8 Tb/s rather than a raw 51.2 Tb/s - silicon that could have been ports went to reduction engines instead.

SHIELD

Self-healing: switches recognise a failed link and route around it without waiting for the subnet manager to recompute.

Recomputing a fabric-wide routing table takes time proportional to the fabric. SHIELD lets neighbouring switches agree on a local detour immediately, so a cable failure degrades a job instead of killing it.

SLO

The latency target a served request is promised - typically a time to first token and a time per output token.

Serving has two clocks and they pull in opposite directions. Time to first token is set by prefill, time per output token by decode, and the lever that improves throughput - a larger batch - makes both worse. Attainment, the fraction of requests meeting the target, collapses well before throughput does.

Spectrum-X

NVIDIA’s Ethernet platform for AI fabrics: switch and SuperNIC co-designed to give Ethernet the properties InfiniBand has by construction.

Adaptive routing decided jointly by the switch and the adapter, telemetry-based congestion control, and lossless operation - the InfiniBand feature list, reached by a different route. Its published performance advantages over standard Ethernet are vendor claims without a stated measured configuration, so this site quotes them as claims.

Straggler

One rank that is slower than the rest, which at a collective barrier makes everyone else that slow too.

Worse than an outright failure in one respect: a failure is detected and handled, while a straggler quietly taxes every step. Because a synchronous collective cannot complete until its slowest participant arrives, a single GPU at 90% speed costs the whole job 10%, no matter how many thousands of healthy peers it has.

structured sparsity

Skipping half the weights in a fixed 2-of-4 pattern, doubling the quoted throughput.

The headline 1.44 EF FP4 figure is a sparse number. Dense FP4 is about half that. Always check which one a benchmark is quoting.

subnet manager

One centralised service that discovers every device in an InfiniBand fabric, assigns addresses and computes every forwarding table.

This is the sharpest architectural difference from Ethernet, which distributes its control plane and lets switches learn. An InfiniBand subnet has a single authority that knows the whole topology, which is what makes deterministic routing and topology-aware collectives possible - and what makes the SM a component whose availability you have to think about.

SuperNIC

NVIDIA’s term for a network adapter built for GPU traffic rather than general server traffic - ConnectX-7 at 400 Gb/s, ConnectX-8 at 800.

One per GPU rather than one per server, which is what makes rail alignment possible and what makes per-GPU egress the meaningful unit of scale-out bandwidth.

SuperPOD

A reference cluster design: 8 NVL72 racks, 576 GPUs, joined by InfiniBand or Spectrum-X Ethernet.

Tensor parallelism

One layer’s matrices are split across GPUs, so a single matrix multiply is executed by several GPUs together.

The finest-grained split, and the most communication-hungry: partial results must be reduced within every layer, twice per transformer block, on the critical path. That traffic is why tensor parallelism is kept inside a coherent domain - across a slower fabric it stops paying almost immediately.

Topology-aware scheduling

Choosing which GPUs a job gets based on how they are wired, not just how many are free.

Sixty-four GPUs spread over six racks and sixty-four inside one NVLink domain are the same number and very different machines. Slurm’s block plugin models the domain explicitly and packs jobs into segments; the older tree plugin optimised for queue time and would fragment a job across leaf switches to start it sooner.

UQD

Universal Quick Disconnect - a dripless blind-mate coupling that connects a tray to the coolant manifold as it is inserted.

Blind-mate matters operationally: a technician slides a tray in from the front and it makes both its data and its liquid connections without anyone touching a hose.

verbs

The de-facto RDMA programming interface: post a work request, poll a completion, never enter the kernel on the data path.

Verbs give three things at once - OS bypass, zero copy, and offload of packet processing to the adapter. Everything above it, including NCCL and MPI, is ultimately posting work requests to queue pairs.

virtual lane

One of several independently buffered and independently credited channels sharing a physical link.

Credits are issued per virtual lane, so a lane whose receive buffers are full stops while the others keep moving. This is what stops one saturated traffic class from freezing the whole link, and it is the mechanism underneath InfiniBand quality of service.