Serving as a Fleet

A benchmark is a coach party: everyone arrives together, wants the same thing, and leaves together. Real traffic is a walk-in restaurant. People turn up when they turn up, one orders a coffee and the next a seven-course meal, and the kitchen that was perfectly efficient for the coach party now has a queue out of the door.

The original way to serve a model was to collect requests into a batch, run the batch to completion, and start the next one. The failure is straightforward and was named precisely by the Orca paper: under request-level batching, "requests that have finished earlier than other requests in a batch cannot return to the client, while newly arrived requests have to wait until the current batch completely finishes".^{Yu et al., OSDI ’22}

Since output lengths vary by more than an order of magnitude, a batch runs at the pace of its longest member. A request that needed twenty tokens waits for one that needed two thousand.

Orca's fix is iteration-level schedulingContinuous batchingAdmitting and retiring requests between decode steps rather than running a batch to completion.: the scheduler "invokes the execution engine to run only a single iteration of the model on the batch",^{Yu et al., OSDI ’22} so membership is reconsidered between every decode step. A finished request leaves immediately and a new one joins on the next token. The industry renamed this continuous or in-flight batching, and it is the single most important thing an inference server does.

It came with a second idea that is less discussed and equally necessary. Batching an attention operation across requests at different sequence positions is not a well-defined matrix operation, so Orca applies selective batching - "batching only to a selected set of operations",^{Yu et al., OSDI ’22} keeping the dense layers batched and handling attention per request.

Serving is judged on two clocks that pull in opposite directions. Time to first token is set by prefill and by however long the request queued. Time per output token is set by decode. The lever that improves throughput - a bigger batch - makes both worse, because decode is bandwidth-bound and a larger batch shares the same 8 TB/sMemory bandwidth per GPU NVIDIA between more sequences.

So a fleet has an SLOSLOThe latency target a served request is promised - typically a time to first token and a time per output token., typically a first-token target and a per-token target, and the number to watch is attainment: the fraction of requests that met it. The figure below runs a standard queueing model over a fixed pool. Throughput rises and then flattens, politely. Attainment falls off a cliff well before that.

Disaggregation splits prefill and decode onto separate GPUs, which is the right decision for the reason the previous chapter gives - the two phases want different hardware characteristics. It also creates a new operational problem that only appears at fleet scale: the correct ratio between the pools depends on the traffic mix, and the traffic mix changes during the day.

A shift towards long documents and short answers loads prefill. A shift towards short prompts and long generations loads decode. Neither pool can help the other, so a fleet sized for the morning mix is mis-sized by the afternoon, and rebalancing means moving model weights between GPUs.

This is where KV cacheKV cacheThe stored keys and values from every previous token, re-read on each decode step. management stops being an implementation detail. Every in-flight request holds a cache proportional to its context, so the batch size a pool can sustain is set by memory, not by arithmetic - and admitting one more request can evict another's cache, forcing a recomputation that looks exactly like a latency spike with no cause. Prefix caching cuts the other way, making repeated system prompts nearly free, which is why serving the same application at scale is cheaper per token than the benchmark suggests.

The honest summary is that a served fleet has more state than a training job does. A training run is one long computation with a checkpoint; a fleet is thousands of short ones sharing a memory budget, and almost every performance surprise in serving traces back to that budget rather than to the model.