Every previous chapter split one thing. A real training run splits all of them at once, and the interesting engineering is in how they compose — because they don't compose freely. Each strategy has a communication signature, each link in the cluster has a bandwidth, and the whole job of designing a parallel layout is matching the heaviest communication to the fastest links. This is the chapter the source notes don't have, and it's where everything so far becomes a single decision.
The device mesh
Think of your GPUs not as a flat list but as a multi-dimensional grid — a device mesh — where each axis is one parallelism strategy. A 512-GPU run might be arranged as:
Every GPU has coordinates and belongs to three groups at once: a tensor-parallel group of 8, a pipeline-parallel group of 8, a data-parallel group of 8. Which strategy gets mapped to which physical axis of the cluster is the entire decision, and it follows one rule.
The one rule: match traffic to bandwidth
Rank the strategies by communication intensity, and rank the links by speed, and line them up. From the bandwidth hierarchy in chapter 1 — NVLink inside a node at ~450 GB/s, InfiniBand between nodes at ~50 GB/s, roughly 9× slower — the assignment writes itself:
| Strategy | Communication | Frequency | Place it on |
|---|---|---|---|
| Tensor | all-reduce, huge | 4× per layer, unhideable | NVLink (in-node) |
| Expert | all-to-all, data-dependent | 2× per MoE layer | NVLink, spilling to fast IB |
| Context | ring KV, hideable | per attention | in-node or fast IB |
| Pipeline | point-to-point activations | once per microbatch/stage | tolerates IB |
| Data | gradient all-reduce, hideable | once per step | tolerates IB (slowest OK) |
Tensor parallelism always goes innermost, on NVLink, because chapter 8 showed its all-reduces sit unhideable in the critical path — put them on a slow link and the tensor cores starve. Data parallelism goes outermost, across the most nodes on the slowest links, because its one gradient all-reduce per step hides almost entirely behind the backward pass and is bandwidth-constant in the number of GPUs (chapter 7's ring result). Pipeline sits in between: its point-to-point sends are light and infrequent enough to cross node boundaries, which is exactly why you reach for it once tensor parallelism has filled the node.
Get this backwards — data parallelism inside the node, tensor parallelism across nodes — and you can lose more than half your throughput to exposed communication while every profiler tells you the GPUs are "busy." They are: busy waiting on the network.
Working it for the anchor models
Llama 3 8B. The fixed state is 120 GiB (chapter 2), just over one H100. You do not need the heavy machinery — the model nearly fits. FSDP/ZeRO-3 across a modest number of GPUs shards the 120 GiB comfortably, activation checkpointing or flash attention handles the elastic term, and you're done. No tensor or pipeline parallelism required. The right layout is the simplest one that fits, and reaching for 4D parallelism on a model this size is a mistake that costs you communication for nothing.
Llama 3 70B. Now the fixed state is ~1 TB — 14 H100s' worth — and it genuinely needs model parallelism. A typical layout: TP=8 within each node (the 70B's large matrices split cleanly across 8 NVLinked GPUs), PP across several nodes to fit the depth, and DP across the remaining GPUs to scale throughput and consume the full batch. Tensor parallelism handles the per-layer memory, pipeline handles the layer count, data parallelism handles the tokens.
Llama 3 405B pushes every axis harder and adds the pipeline rebalancing from chapter 9, but the shape is the same: TP innermost on NVLink, PP across nodes, DP outermost.
MFU: whether any of it worked
You designed a layout; the scoreboard for whether it was a good one is Model FLOPs Utilization from chapter 1. Every parallelism decision shows up in that single number, because everything that isn't the model's own math — every exposed all-reduce, every pipeline bubble, every load-imbalanced expert, every recomputed activation — is time the tensor cores spend not doing useful FLOPs, and MFU is exactly the fraction they do.
That's why 35–45% is the realistic target and Llama 3 405B landed at 38–43% at 16k-GPU scale.Grattafiori et al., The Llama 3 Herd of Models, 2024, §3.3. The gap from 100% is not waste to be embarrassed about — it's the irreducible cost of splitting a computation across sixteen thousand chips. The remaining ~60% isn't incompetence; it's the sum of every trade in this book, made as well as anyone currently knows how. When you improve a layout, MFU is the number that moves, and if it doesn't move, the change didn't matter regardless of how clever it was.
The order the decision actually gets made in is bottom-up from the hardware, not top-down from the model. You start from the node topology — how many GPUs share an NVLink island, how the islands connect — because that fixes the innermost axis before you've said anything about the model: tensor parallelism can be at most the island size, full stop. Then the memory budget from chapter 2 tells you how much more splitting you need beyond that, which sets the pipeline depth. Data parallelism is whatever's left over. MFU comes last, as the verdict, not the input — you design the layout from constraints and then measure whether the constraints let you reach a good number.
And the thing that most often keeps you from reaching it is dumber than any of the theory in this book. It's rarely the parallelism math being wrong; it's a data loader starving the GPUs, a single slow node dragging every collective, a logging or checkpoint hook that blocks the step, a kernel that quietly fell back to a slow path. The parallelism strategy is the part you reason about carefully and usually get right. The throughput you actually lose tends to leak out through the plumbing — which is why the last step below is "measure, find the worst axis, fix that one thing" rather than "derive the optimal layout and trust it."
How to actually approach a new model
A rough decision order that respects everything above:
- Does it fit with data parallelism alone (FSDP/ZeRO)? If yes, stop — you've got the cheapest communication profile there is. This is most models under ~10B.
- If not, add tensor parallelism up to the node boundary (TP ≤ 8). This handles per-layer memory with the fastest links you have.
- Still doesn't fit? Add pipeline parallelism across nodes, and mind the embedding/loss imbalance at the ends (chapter 9).
- Long context? Add context parallelism on the sequence dimension.
- MoE? Add expert parallelism, across nodes since all-to-all tolerates it better than TP's all-reduce, and tune the router's balance.
- Data-parallel over everything that's left, on the slowest links, to scale throughput.
- Measure MFU. Find the least-balanced axis. Fix that one thing. Repeat.
The ordering is not arbitrary — it's cheapest-communication-first, and it's the same logic as the whole book: spend the budget you have the most of, on the constraint that's actually binding.