Precision & Quantization

Quantization is the one technique in this book that spends all three budgets at once. Fewer bits per weight means less memory, fewer bytes moved per weight read, and — on hardware with low-precision tensor cores — more FLOP/s. Against the chapter 1 roofline it's a rare move that pushes you rightward and upward simultaneously.

What it spends in return is precision, and the entire craft of quantization is in spending as little of it as possible. The naive version is trivial and the naive version breaks at scale, in a specific and interesting way. So this chapter is mostly about the failure and the fixes.

What the bits actually buy you#

First, precision formats, because "16-bit" hides the decision that matters. A floating-point number splits its bits between an exponent (dynamic range, how big and small it can go) and a mantissa (precision, how many significant figures within that range).

The bf16-vs-fp16 choice is the instructive one, and it's why training switched. Both are 16 bits. fp16 spends them on precision (10 mantissa bits) at the cost of range; bf16 keeps fp32's entire exponent and sacrifices precision (7 mantissa bits). For training, range wins decisively — gradients span many orders of magnitude and small ones underflowing fp16's range to zero was a constant source of instability. bf16's wider range is exactly why it retired loss scaling, the fp16-era hack of multiplying the loss by a large constant to drag small gradients up into representable territory before they vanished.This is the same master-weights argument from chapter 2: bf16 has ~2 decimal digits of precision, which is why the optimizer step still runs in fp32.

Two regimes follow, and they're genuinely different problems:

• Weights and activations in float (bf16/fp16, sometimes fp8) — this is training and native inference precision. Values are live, gradients flow.
• Weights quantized to low-bit integers (int8, int4) for inference only — a trained model compressed after the fact. No gradients, and a different set of problems.

The float story is mostly chapter 2. This chapter is the integer story, because that's where the interesting failure lives.

The trivial version#

Take a tensor of weights and map its range onto the integer grid. Two choices:

Symmetric (abs-max): find $\alpha = \max|w|$, set the scale $s = \alpha / (2^{b-1}-1)$, and round: $q = \mathrm{round}(w/s)$, clamped to the integer range. Dequantize with $\hat{w} = s \cdot q$. One number, the scale, per tensor.

Asymmetric (min/max): map $[\min, \max]$ onto the full integer range with a zero-point offset. Uses the range more efficiently for skewed distributions (post-ReLU activations especially), at the cost of a second stored number.

The error on any weight is bounded by half a step, $s/2$, which for a well-behaved bell-shaped distribution is uniform quantization noise a well-trained network shrugs off. In 2021 this looked basically solved.

Then models got bigger.

The thing that breaks: emergent outliers#

Here is the whole problem in one picture. Move the outlier slider and watch what a single extreme value does to every other weight:

At scale — Dettmers et al. pin the transition at around 6.7B parameters — a small number of feature dimensions in the activations start carrying values 20× or more larger than the rest.Dettmers et al., LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale, NeurIPS 2022. The outliers are systematic: they occur in specific feature dimensions across many tokens, not at random. These "emergent features" are not noise — ablating them wrecks the model, so they must be preserved. But abs-max quantization keys the scale to the largest magnitude present. One 20σ outlier drags $\alpha$ up 20×, which coarsens the step $s$ by 20× for all the ordinary weights, and the bulk of the distribution collapses onto a handful of integer levels. The error you see in the widget's bulk jumps even though only one value was extreme.

This is why naive int8 falls off a cliff at exactly the scale you most want to compress. Every real quantization method is, at heart, a different answer to "what do we do about the outliers?"

The four answers that matter#

LLM.int8() — isolate them. Decompose the matmul: run the ~0.1% of outlier dimensions in fp16, everything else in int8, and sum. Essentially lossless, but the mixed-precision path is slow — you're paying for two datatypes and a scatter/gather. Great for "fit the model at all," not for maximum throughput.

SmoothQuant — migrate the difficulty. Activations have the outliers; weights are smooth and easy. So shift the burden: scale activations down by a per-channel factor $s$ and scale the corresponding weight columns up by the same $s$, leaving the product $XW$ mathematically unchanged. Now both are moderately hard instead of one being impossible, and both quantize to int8 cleanly.Xiao et al., SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models, ICML 2023. The migration strength is a tunable α, usually ~0.5. It's a genuinely elegant reframing — you don't fight the outliers, you redistribute them into a channel that can absorb them.

GPTQ — quantize with a conscience. Round weights one at a time, and after each rounding adjust the not-yet-quantized weights in the same layer to compensate for the error just introduced. The adjustment uses second-order information — the Hessian of the layer's reconstruction loss, approximated from a small calibration set. It's the Optimal Brain Surgeon idea made fast enough for billion-parameter layers, and it's what makes reliable int4 possible.Frantar et al., GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers, ICLR 2023.

AWQ — protect what matters. Not all weights are equal; the ~1% whose channels see large activations dominate the output. AWQ identifies them by activation magnitude (not weight magnitude — the key insight) and scales them to protect their precision before quantizing. Cheaper than GPTQ, no backprop, and often better on instruction-tuned models.Lin et al., AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration, MLSys 2024.

All four are post-training quantization (PTQ): take a finished model, compress it, no retraining. That's the cheap common case, and for int8 it's nearly free. For int4 you want GPTQ or AWQ; for int8 SmoothQuant or plain per-channel usually suffices.

When PTQ isn't enough: training through the round#

For very low bit-widths, or when you can afford the compute, you fold quantization into training itself — quantization-aware training (QAT). The forward pass simulates the full quantize-dequantize round trip, so the network learns weights that survive it, and the loss reflects the quantization error directly.

Which raises an immediate problem: rounding has a derivative of zero almost everywhere and undefined at the steps. Backprop through it kills every gradient. The fix is the straight-through estimator — in the backward pass, pretend the rounding was the identity function:

Gradient of 1 inside the representable range, 0 outside (a weight clipped to the boundary shouldn't be pushed further out). It's a lie about the true derivative, and it works — the forward pass sees the real quantization error, the backward pass gets a usable signal, and the mismatch washes out over training.Bengio et al., Estimating or Propagating Gradients Through Stochastic Neurons, 2013, is the origin; the framing here follows the quantization literature (e.g. Jacob et al., 2018).

class FakeQuant(torch.autograd.Function):
    @staticmethod
    def forward(ctx, w, scale, qmin, qmax):
        q = torch.clamp(torch.round(w / scale), qmin, qmax)
        ctx.save_for_backward(w, scale.new_tensor(qmin), scale.new_tensor(qmax), scale)
        return q * scale                      # real quantization error, forward
 
    @staticmethod
    def backward(ctx, grad_out):
        w, qmin, qmax, scale = ctx.saved_tensors
        # Straight-through: pass the gradient where we didn't clip, kill it where we did.
        mask = (w >= qmin * scale) & (w <= qmax * scale)
        return grad_out * mask, None, None, None

QAT gets you usable int4 and even int2 where PTQ falls apart, at the cost of a training run. The frontier — QLoRA and its descendants — combines a 4-bit frozen base model with small trainable adapters, making fine-tuning of a 70B model possible on a single GPU.Dettmers et al., QLoRA: Efficient Finetuning of Quantized LLMs, NeurIPS 2023 — 4-bit NormalFloat, double quantization, and paged optimizers.

Don't forget the KV cache#

One quantization win is easy to miss because it's not about weights at all. From chapter 5, the KV cache is often the binding memory constraint in serving, and it quantizes independently of the model. fp8 or int8 KV cache is a straight 2× on the largest term in your serving memory budget, it composes with every attention variant, and because the cache is read fresh from HBM every decode step, halving its bytes directly improves decode's arithmetic intensity. It is frequently the single easiest win available to an inference service, and the playground's fp8 setting is a fair proxy for how gently it degrades.

What to actually reach for#

The through-line: quantization is cheap in the easy case and the easy case doesn't survive scale. Everything past int8 is a different strategy for keeping a handful of outliers from ruining the compression for everyone else.