Scheduling and Placement

Booking a table for eight is not the same as booking eight seats. A restaurant with thirty free chairs scattered around the room cannot seat your party, and telling you it is only 60% full does not help. Tightly coupled jobs are parties, not individuals.

Two properties make these jobs awkward to schedule. First, they are indivisible in time: gang schedulingGang schedulingStarting all of a job’s ranks together or not at all, because a collective cannot proceed with some of them missing. means all ranks start together or none do, because a collective cannot complete with participants missing. Half an allocation now and half in an hour is worth nothing.

Second, they are indivisible in space, or nearly. Chapter 1 establishes that the tensor-parallel group has to sit inside one NVLink domain. So the request is not "72 GPUs" but "72 GPUs arranged such that these particular 8 are mutually coherent" - a constraint on adjacency, which a scheduler counting free capacity cannot see at all.

Slurm's history here is instructive. The older tree plugin optimised for queue time, and its own rationale says so: "A job split across multiple leaf switches might run slightly slower than under one single leaf switch, but the tradeoff between start time and performance is generally considered acceptable."^NVIDIA That is a defensible trade right up until the penalty for spreading out stops being slight.

On this hardware it is not slight. "The new topology/block plugin was introduced in the Slurm 23.11 release to support rack-scale architectures such as GB200 NVL72",^NVIDIA modelling the domain explicitly and packing jobs into blocks. One domain is eighteen nodes, and for jobs above that size NVIDIA recommends a 16 nodesSlurm block-plugin segment size One NVL72 domain is 18 nodes. For jobs larger than that, NVIDIA recommends "--segment=16 instead" of 18, so an allocation is assembled from blocks that are each findable and each large enough to hold a full tensor-parallel group. NVIDIA rather than eighteen - so topology-aware schedulingTopology-aware schedulingChoosing which GPUs a job gets based on how they are wired, not just how many are free. is the argument of Part I restated as a scheduler policy.

Turn the churn up under topology-aware placement and the utilisation figure falls while jobs start being rejected for shape. This is not a bug in the policy - it is the policy. FragmentationFragmentationFree GPUs that are the wrong shape - enough capacity in total, but not enough of it adjacent. is the price of insisting on adjacency, and a cluster can sit at 85% allocated while refusing a 72-GPU job because no single domain has 72 free.

The capacity-only policy has the opposite failure. It never rejects anything it has room for, and it produces jobs whose tensor-parallel groups straddle racks - which, by the arithmetic of chapter 7 of Part I, drops the bandwidth available to the chattiest dimension in the model from 900 GB/sNVLink egress per GPU, one direction Half of the 1.8 TB/s bidirectional headline. This is the figure to compare against a scale-out line rate, because a NIC’s quoted rate is also one direction. NVIDIA to 100 GB/sConnectX-8 egress per GPU 800 Gb/s. Still 9× less than in-rack NVLink egress. NVIDIA . The job runs. It runs badly, and the cluster's dashboards look excellent.

Which makes utilisation an actively misleading metric for these clusters, and it is the one most operators report. A fleet at 95% allocation running jobs at half their achievable MFU is doing less work than one at 80% running them well. The measure that survives contact with this problem is the goodput of the previous chapter applied to the fleet: not how many GPUs are busy, but how much finished work they produce.

Draining and repacking recovers the shape, at the cost of throughput while it happens. That is the real decision a scheduler policy encodes, and reasonable operators land in different places on it depending on whether their queue is dominated by a few enormous jobs or many small ones.