The Three Budgets
Roofline, arithmetic intensity, and the one ratio that explains why decoding and training feel like different machines.
LLMs, efficiently
Memory, compute, and bandwidth are the only three budgets there are. Every technique for training and serving large language models — checkpointing, flash attention, GQA, quantization, ZeRO, tensor and pipeline parallelism, speculative decoding — spends one to buy back another. This is that ledger, derived rather than listed.
Built on the outline of Gauri Gupta's LLM optimization notes — an excellent index of this material that this book tries to add depth to. Every technical claim here is written from the primary sources, cited throughout.
Where the Memory Goes
Parameters, gradients, optimizer state, activations — and why an 8B model needs 128 GB before it stores a single activation.
Activation Checkpointing
The autograd tape, sublinear memory, and why selective recompute beats the full version by an order of magnitude.
Flash Attention
Derive online softmax first and tiling falls out of it. An exact algorithm, not an approximation.
The KV Cache
MHA, MQA, GQA, MLA, paged attention, prefix caching — one problem and six responses to it.
Precision & Quantization
Bit layouts, emergent outliers, GPTQ, AWQ, SmoothQuant, and training through a non-differentiable rounding step.
Data Parallelism & ZeRO
Ring all-reduce from first principles, then ZeRO's three stages accounted in both memory and network traffic.
Tensor Parallelism
Why Megatron splits column-then-row, what sequence parallelism buys, and why this never leaves the node.
Pipeline Parallelism
The bubble, derived. GPipe versus 1F1B, weight staleness, zero-bubble schedules, and what Llama 3 actually changed.
Context & Expert Parallelism
Splitting the sequence, splitting the FFN — ring attention, routing, capacity factors, and load balance.
Composing 4D Parallelism
How DP, TP, PP, CP, and EP actually compose on a real cluster, and how to pick the layout.
Serving
Continuous batching, chunked prefill, disaggregation, and why speculative decoding is provably lossless.
What Lost, and Why
An honest postmortem on efficient-attention research, plus the full bibliography.