The Three Budgets

There's a version of this subject that's just a vocabulary list. Flash attention, GQA, ZeRO stage 3, speculative decoding, pipeline bubbles. You can learn to say all of it in an afternoon and still have no idea which one to reach for when your training run is at 22% MFU, or why your inference server falls over at 40 concurrent requests when the GPU claims to be 4% busy.

The vocabulary list is a symptom of skipping the part that makes it cohere. Because almost everything in this book is the same move, executed against different constraints:

You have three budgets — memory, compute, and bandwidth. Every technique here spends one to buy back another. None of them are free, and which trade is worth making depends entirely on which budget you're currently out of.

Activation checkpointing buys memory with FLOPs. Grouped-query attention buys bandwidth with model capacity. ZeRO stage 3 buys memory with network traffic. Speculative decoding buys latency with wasted compute. Quantization buys all three with precision. Once you can see the trade, the techniques stop being a list to memorize and start being a decision you can actually reason about.

So before any of them: what are the budgets, and how do you tell which one you're out of?

The machine you are actually programming#

Start with the hardware, because the hardware is where the constraint lives.

An H100 SXM does about 989 TFLOP/s of dense bf16 matrix multiply.Dense, not the "with sparsity" figure that doubles it in marketing material and applies to approximately nothing anyone trains. Assume every FLOP/s number in this book is dense unless I say otherwise. Its HBM3 stack delivers about 3.35 TB/s.

Those are the two numbers on the spec sheet everyone quotes. The number that actually matters is their ratio:

Read that as a demand the machine makes of you. For every byte you pull out of HBM, you had better find about 295 floating-point operations to do with it. Find fewer, and the tensor cores sit idle waiting on memory — you are paying for a supercomputer and operating a memory controller. This threshold is the ridge point, and it is the single most useful number in this entire book.

It is also getting worse, which is the part people miss:

From A100 to H100, compute went up 3.2× and bandwidth went up 1.6×. The ridge point nearly doubled. Every hardware generation makes arithmetic cheaper relative to data movement, which means every generation makes the memory-bound parts of your workload relatively worse even as they get absolutely faster. H200 is interesting precisely because it's the exception — same compute die, much more bandwidth, so it's an H100 that's better at the memory-bound half of the job.

This is the memory wall, and it isn't a metaphor. It's a ratio you can look up, and it's climbing.

Arithmetic intensity#

The corresponding property of your workload is arithmetic intensity:

Two things about this definition deserve more attention than they usually get.

The denominator is HBM traffic, not total data touched. A value that gets read once into SRAM and reused a thousand times from there counts once. This is not a technicality — it is the optimization. Tiling a matmul doesn't change its FLOPs or its logical data dependencies; it changes how many times a byte crosses the HBM boundary, which changes $I$, which is the only thing the hardware rewards. Flash attention is the same insight applied to attention, and we'll do it properly in chapter 4.

Intensity is a property of an algorithm at a given shape, not of an algorithm. The same matmul kernel has wildly different intensity depending on the matrix dimensions. Consider $A \in \mathbb{R}^{m \times k}$ times $B \in \mathbb{R}^{k \times n}$ in bf16:

If all three dimensions are large, this is roughly $\frac{1}{1/m + 1/n + 1/k}$ — it grows with the smallest dimension. A $4096^3$ matmul has intensity around 1365, comfortably compute-bound. But set $m = 1$, a matrix times a vector:

Intensity 1, regardless of how big $n$ and $k$ are. You read each weight once and do exactly one multiply and one add with it. Two FLOPs, two bytes.

Hold onto that, because it's about to explain your entire inference bill.

The roofline#

Put the two together and you get the roofline model.Williams, Waterman & Patterson, Roofline: An Insightful Visual Performance Model for Multicore Architectures, CACM 2009. It predates all of this by fifteen years and remains the right first tool. The best performance any kernel can achieve is:

On log-log axes this is a diagonal line (bandwidth-limited, slope 1) that flattens into a horizontal ceiling (compute-limited) at the ridge point. Every kernel you write is a dot under that roof. Where it sits tells you what to do about it:

• Left of the ridge (memory-bound). Buying a faster GPU buys you almost nothing; the FLOPs were never the problem. Reduce bytes moved, or increase reuse. Fuse kernels, tile better, quantize, batch harder.
• Right of the ridge (compute-bound). You're using the machine as intended. Now the wins are lower precision, better kernels, or more GPUs.
• Far below the roof on either side. Something else is wrong — launch overhead, poor occupancy, a synchronization stall, exposed communication. The roofline can't see those, which is a real limitation I'll come back to.

Here it is with real models on real hardware. Move the batch size slider and watch what happens to the decode point:

Two workloads, one model, three orders of magnitude apart#

The reason inference and training feel like different disciplines is that they sit on opposite sides of that ridge. Let's do the arithmetic on Llama 3 8B, which will be our running example for the whole book.

Decoding one token#

Generate a single token for a single sequence. What does the GPU actually do?

It reads all 8.03B parameters out of HBM — 16.06 GB — and performs one multiply-accumulate per parameter, so $2 \times 8.03\text{B} = 16.06$ GFLOP. Every one of those GEMMs has $m = 1$. It's the matrix-vector case:

Exactly 1. Not approximately — exactly, and independently of model size, because bf16 weights are two bytes and a MAC is two FLOPs. Against a ridge point of 295, this is a 295× shortfall. Concretely:

• Time to move the weights: $16.06\ \text{GB} \div 3.35\ \text{TB/s} = 4.79$ ms
• Time to do the math: $16.06\ \text{GFLOP} \div 989.4\ \text{TFLOP/s} = 16.2$ µs

The tensor cores work for 16 microseconds and then wait for 4.8 milliseconds. Utilization: 0.34%. Your ceiling is about 209 tokens/second, and no kernel optimization will move it, because you are not running a computation — you are streaming 16 GB through a chip and occasionally multiplying.

This is the whole reason single-stream decoding is slow, and it reframes every inference technique in this book as an answer to one question: how do we get more FLOPs out of each byte of weight we're forced to read?

• Read fewer bytes per weight → quantization (ch. 6)
• Do more work per read → batching (ch. 12)
• Read fewer weights per token → MoE (ch. 10)
• Produce more than one token per read → speculative decoding (ch. 12)
• Stop the KV cache from becoming the new bottleneck once you've fixed the weights → GQA, MLA, paged attention (ch. 5)

That list isn't five unrelated tricks. It's five ways to attack one fraction.

Prefilling a prompt#

Now process a 2048-token prompt in one shot. Same model, same weights, same single read of them — but now 2048 tokens of work ride on that read. The GEMMs have $m = 2048$ instead of 1.

The second term is attention, and notice it's only 3.2% of the total at this length — a useful corrective to the instinct that attention dominates transformers. It doesn't, until the sequence gets long. (The quadratic term grows with $S^2$ while the linear term grows with $S$, so they cross eventually; at 2048 we're nowhere near it.)

Bytes moved: the same 16.06 GB of weights, plus 268 MB of KV cache written.

2082 against a ridge of 295. Solidly compute-bound, 7× past the ridge. This one is a real computation, and it runs at something close to the speed the GPU was sold at: 34.0 TFLOP ÷ 989.4 TFLOP/s ≈ 34.4 ms.

Now put the two side by side, because this comparison is the point of the whole chapter:

Same weights. Same GPU. Same kernels, even. A 285× difference in cost per token, entirely because of the shape of the matrices.

I want to be clear that this is not a subtle effect you find with a profiler. It's the dominant fact about serving transformers, and if a design discussion about inference isn't grounded in it, the discussion is decorative.

The third budget: bandwidth that isn't HBM#

I've been treating "bandwidth" as HBM bandwidth, which is right for a single GPU. The moment you use more than one, a second, much slower tier appears — and the training half of this book is largely about it.

Rough hierarchy on an H100 node:

Each step down is roughly an order of magnitude, and that hierarchy dictates essentially every parallelism decision in chapters 7 through 11. Tensor parallelism synchronizes twice per transformer block, so it has to live on NVLink and never crosses a node boundary. Pipeline parallelism only passes activations at stage boundaries, so it tolerates InfiniBand. ZeRO stage 3's extra 50% communication volume is affordable inside a node and often ruinous across many. These aren't arbitrary conventions — they're this table.

Keeping score: MFU#

If the roofline is the diagnosis, Model FLOPs Utilization is the scoreboard. Training a dense transformer costs about $6N$ FLOPs per parameter per token — $2N$ forward, $4N$ backward, since the backward pass computes gradients with respect to both inputs and weights.The standard accounting from Kaplan et al., Scaling Laws for Neural Language Models, 2020. It ignores attention's quadratic term, which is fine below ~8k context and misleading above it. So:

for $D$ tokens in $T$ seconds on $G$ GPUs.

The crucial detail is the word Model. MFU counts only the FLOPs the model mathematically requires. If you're using activation checkpointing you're also running a second forward pass whose FLOPs the model didn't ask for; counting those gives you Hardware FLOPs Utilization instead. HFU is always at least MFU, and the gap between them is precisely the tax you're paying for recomputation — which makes the pair of them the cleanest way to evaluate chapter 3.

For calibration: large dense pretraining runs on H100s land somewhere around 35–45% MFU, and Meta reported 38–43% for Llama 3 405B across up to 16k GPUs.Grattafiori et al., The Llama 3 Herd of Models, 2024, §3.3. If you're at 15%, something structural is wrong and this book probably names it. If someone claims 70% on a dense model, ask what they're counting.

The number nobody tells you is how fragile MFU is in practice. A run that holds 40% for a week can quietly slip to 32% and the cause is almost never the model — it's the plumbing around it. The usual suspects, roughly in order of how often they're the culprit: a data loader that can't keep the GPUs fed, so they stall waiting on input; a straggler node whose slightly-slow NIC makes every all-reduce wait on it; checkpoint saves that block the training step instead of overlapping; a stretch of unusually long sequences inflating the attention term; or a silent fallback to a slower kernel after a library upgrade. The reason this matters for a book about the roofline is that none of these show up on the roofline — every one of them is the "far below the roof" case, where the dot sits under the line for reasons the model can't see. MFU is the tripwire that tells you to go looking; the roofline tells you where not to bother.

Where the roofline lies to you#

I'd rather introduce the limitations now than have you discover them at 2am.

It assumes perfect overlap. The model says time is $\max(\text{compute}, \text{memory})$. Real kernels often get closer to the sum, especially when a dependency prevents prefetching the next tile.

It has no notion of latency or occupancy. A kernel can sit far below the roof because it launches 40 microseconds of work per launch, or because register pressure caps it at two resident blocks per SM. The roofline shows a dot below the line and shrugs.

It ignores caches. L2 on an H100 is 50 MB, which is enough to hold real working sets. Traffic served from L2 never touches HBM, so measured intensity can beat the analytical estimate.

It says nothing about correctness. Quantization moves you left along the x-axis and also degrades your model. The roofline will happily show you a beautiful dot for a 2-bit model that outputs garbage.

Used as a first tool, it's unreasonably good: it tells you which of two qualitatively different regimes you're in, and therefore which half of this book to read. Used as a final tool, it will mislead you. Profile before you believe anything.

What's next#

We now have units. From here, every technique gets reported in them: what it costs in memory, what it costs in FLOPs, what it costs in bytes moved, and which of those you were out of anyway.

The next chapter spends the memory budget in detail — where all 80 GB of an H100 actually goes during training, why an 8B model needs 128 GB before it stores a single activation, and why activations turn out to be the term that matters most.