Serving

Training is throughput. Serving is a harder problem because it's two problems with opposite characters, run against a stream of unpredictable requests arriving whenever they like, each demanding a low latency you promised in an SLA. Everything in this chapter follows from a single fact established back in chapter 1: prefill and decode sit on opposite sides of the roofline, and a good serving system stops pretending they're the same workload.

Two workloads wearing one model#

Recall the numbers. Prefill processes the whole prompt at once — big matmuls, arithmetic intensity in the thousands, compute-bound, ~100% of peak. Decode generates one token at a time — matrix-vector products, arithmetic intensity 1, memory-bound, 0.3% of peak. Same weights, same GPU, a 285× difference in cost per token.

Design consequences fall straight out:

• Decode is memory-bound, so batching is nearly free. Running 32 sequences through decode costs almost the same wall-clock as running one, because the bottleneck is reading the weights, and you read them once for the whole batch. Serving throughput is almost entirely a question of how large a decode batch you can assemble and keep full.
• Prefill is compute-bound, so batching barely helps and a long prompt can monopolize the GPU, stalling everyone else's decode.

The three big serving techniques are each a direct response to one of these.

Continuous batching: never wait for the batch#

Static batching — collect $N$ requests, run them together to completion — wastes enormous throughput, because requests finish at different times. A batch of 32 where one request generates 2000 tokens and the rest generate 50 leaves 31 slots idle for the long tail of the run, and no new request can join until the whole batch drains.

Continuous batching (a.k.a. in-flight batching) operates at the granularity of a single decode step instead of a whole request.Yu et al., Orca: A Distributed Serving System for Transformer-Based Generative Models, OSDI 2022, introduced iteration-level scheduling; vLLM and TGI made it standard. After every step, finished sequences leave the batch and waiting requests join immediately. The batch is re-formed each iteration, so the GPU is always as full as the queue allows. Combined with paged attention's non-contiguous KV allocation — which is what lets a new sequence slot into a batch mid-flight without a contiguous buffer reservation — this is the single largest throughput win in modern serving, often several-fold over static batching.

Chunked prefill and disaggregation: keep decode moving#

The remaining problem is prefill stalling decode. When a request with an 8k prompt arrives, its prefill can take tens of milliseconds — during which every other user's token generation freezes. Two answers, increasingly aggressive:

Chunked prefill. Break a long prefill into fixed-size chunks and interleave them with ongoing decode steps, so decode never stops for more than one chunk.Agrawal et al., Sarathi-Serve: Taming Throughput-Latency Tradeoff in LLM Inference, OSDI 2024. It also improves prefill's own efficiency by fusing a compute-bound prefill chunk with the memory-bound decode work into one balanced batch — you're using the tensor cores (for prefill) and the memory bus (for decode) at the same time instead of alternating.

Prefill/decode disaggregation. Run prefill and decode on separate pools of GPUs entirely.Zhong et al., DistServe: Disaggregating Prefill and Decoding for Goodput-Optimized LLM Serving, OSDI 2024. Prefill machines do compute-bound prompt processing; decode machines do memory-bound generation; the KV cache is handed off over the network between them. Now each pool is tuned for one workload — prefill for FLOPs, decode for memory bandwidth and large batches — and neither interferes with the other. It's the logical endpoint of "these are two different workloads": give them two different machines.

Speculative decoding: more than one token per weight read#

Now the technique the source notes give three sentences, missing the one property that makes it remarkable.

The setup follows from decode being memory-bound. A target forward pass reads all the weights to produce one token, and — critically — it could verify several candidate tokens in parallel for almost the same cost, because the cost is the weight read, not the arithmetic. So: have a small, fast draft model guess the next $k$ tokens cheaply, then have the target model check all $k$ in a single forward pass. Accept the longest correct prefix, and you've produced multiple tokens from one target weight read.Leviathan et al., Fast Inference from Transformers via Speculative Decoding, ICML 2023; Chen et al., Accelerating Large Language Model Decoding with Speculative Sampling, 2023 — concurrent, with the same core algorithm.

The tradeoff surface: draft length $k$, per-token acceptance rate $\alpha$ (how often the draft agrees with the target), and the draft's cost relative to the target. Expected tokens per verification step is a geometric series, $(1 - \alpha^{k+1})/(1 - \alpha)$, which trades against the $k$ draft passes you had to run. Explore it:

Push $\alpha$ up and the speedup climbs; push the draft cost up and it collapses. There's an interior optimum for $k$ — too short wastes the free parallel verify, too long spends draft compute on tokens that will be rejected. Typical production numbers land at 2–3×.

Why it's exactly lossless#

Here's the property that ought to make you suspicious, and then convinced. Speculative decoding produces output drawn from precisely the target model's distribution — not approximately, not "close enough," exactly the same distribution as if you'd decoded from the target token by token. A cheap guessing model in the loop, and zero quality cost. How?

The magic is in the acceptance rule. Let $p(x)$ be the target's probability for a token and $q(x)$ the draft's. For each drafted token $x$:

• Accept it with probability $\min\left(1, \frac{p(x)}{q(x)}\right)$.
• On rejection, don't just resample from $p$ — that would bias the result. Sample from the residual distribution $\text{norm}(\max(0, p(x) - q(x)))$, which is the part of $p$ the draft under-weighted.

The claim is that a token emitted by this accept-or-resample procedure is distributed exactly as $p$. The proof is a short case split on the probability that the final token equals some value $x$:

The accepted mass contributes $\min(p, q)$ and the residual contributes exactly the shortfall $p - \min(p, q)$; they sum to $p$ at every $x$. The draft model's quality affects only how often you accept — the speed — and never what you emit — the distribution. That decoupling is the whole idea, and it's why speculative decoding is deployed by default rather than treated as a quality/speed knob. You are not trading accuracy for latency. You are getting latency for free.

The variants worth knowing#

The draft doesn't have to be a separate model, and removing it removes the main operational headache (serving and aligning two models):

• Self-speculation / Medusa — bolt extra lightweight prediction heads onto the target model itself to propose the next few tokens. No second model to serve.Cai et al., Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads, 2024.
• EAGLE — draft in the target's feature space rather than token space, raising the acceptance rate and thus the speedup.
• Prompt lookup / n-gram — for tasks with heavy input copying (summarization, code editing, RAG), "draft" by simply grabbing n-grams from the prompt. Zero model cost, and shockingly effective when the output quotes the input.

Putting a serving stack together#

A modern high-throughput server is the composition of all of the above: paged KV cache for memory efficiency, continuous batching to keep the GPU full, chunked prefill (or full disaggregation at scale) to keep decode responsive, prefix caching (chapter 5) to skip recomputing shared prompts, quantized weights and KV cache (chapter 6) to shrink the reads, and speculative decoding to get multiple tokens per read. Each targets a specific term in the chapter 1 cost model, and they stack because they attack different terms. That's the recurring shape of this whole book: there's no single win, only a stack of budget trades, each one bought with a resource you had to spare.