Quantization for inference

The big idea

A 70-billion-parameter model in 16-bit needs 140 GB just for its weights, which is two 80 GB GPUs before a single user is served. Store each weight in 4 bits instead and the same model needs about 35 GB: it fits on one GPU with room left for the KV cache, and every decode step reads a quarter of the bytes. That is the promise of quantizationStoring numbers such as weights or activations in fewer bits, for example 8 or 4 instead of 16, by mapping them onto a small set of levels with a scale factor..

The catch is that 4 bits can only name 16 different values. Think of a shop that stops printing exact prices and instead labels each item with one of 16 colored stickers, where each aisle has its own price chart for the colors. Most items end up close to their real price. But if one aisle contains a single very expensive item, its chart has to stretch to cover it, and every cheap item in that aisle gets rounded to "almost free".

Every practical quantization method is about managing that trade: how many levels, how many items share one chart, and what to do about the expensive outliers.

How it works

Scales, levels and rounding

The simplest scheme is symmetric round-to-nearest, sometimes called absmax quantization. With $b$ bits you get integer codes from $-L$ to $L$, where $L=2^{b-1}-1$: 127 for 8 bits, 7 for 4 bits. For a group of weights:

Worked example in 4 bits: a group whose largest weight is 0.035 gets $s=0.035/7=0.005$. A weight of 0.012 becomes $q=\mathrm{round}(2.4)=2$, stored as the integer 2 and read back as $2\times 0.005=0.010$. The rounding error is 0.002, and no weight's error can exceed half a step, 0.0025.

Now put one outlier of 0.16 in the same group. The scale becomes $0.16/7\approx 0.023$, and every weight smaller than half of that, about 0.011, rounds to zero. In a group of typical weights, that can be most of them. This is why the number of weights sharing one scale matters as much as the bit width.

Groups: more scales, less damage

Giving each small group its own scale contains the damage: an outlier only stretches the scale of its own group. The cost is the scales themselves. A 16-bit scale per group of 128 weights adds $16/128=0.125$ bits per weight, so "4-bit" is about 4.1 bits; groups of 32 add 0.5 bits. A large study of 3- to 8-bit inference found small block sizes to be one of the few changes that reliably improved the accuracy you get per bit of model size (see In the wild).

The formats: INT8, INT4 and FP8

• INT8: 256 evenly spaced levels. With per-channel or per-group scales, weights barely notice it.
• INT4: 16 levels. Usable for weights with groups and a smarter rounding method (below); too coarse for activations in most setups.
• FP8: an 8-bit floating-point format. Instead of evenly spaced levels, it spends some bits on an exponent, so it covers small and large values with roughly constant relative precision. The common pair of encodings is E4M3An 8-bit floating-point encoding with 4 exponent bits and 3 mantissa bits: more precision, less range. Usually used for weights and activations at inference. (4 exponent and 3 mantissa bits) and E5M2 (5 and 2: more range, less precision). Floating point handles outliers more gracefully than evenly spaced integers, which makes FP8 a popular choice for activations.

Weight-only versus weight and activation quantization

There are two very different things you can quantize, and they speed up different phases.

Weight-only quantizationQuantizing only the stored weights (for example to 4 or 8 bits) while activations and the arithmetic stay in 16-bit. Written W4A16 or W8A16. Saves memory and bandwidth. (written W4A16 or W8A16: 4- or 8-bit weights, 16-bit activations) shrinks what is stored and read. The kernel rescales the codes back to 16-bit on chip and multiplies as usual. From How LLM inference works : single-user decoding is memory-bound, so reading a quarter of the bytes can make each step up to about 4 times faster in the ideal case. Prefill, which is compute-bound, gains little, because the multiplications are the same size.

Weight and activation quantizationQuantizing both the weights and the activations flowing between layers, for example W8A8 (INT8) or FP8, so the matrix multiplications themselves run in low precision. (W8A8 in INT8, or FP8) also quantizes the inputs to each matrix multiply, so the multiply itself runs on low-precision tensor cores. That speeds up compute-bound work too: prefill, and decoding at large batch sizes. It is harder, because activations are computed fresh for every token and contain outliers.

Activation outliers

Activations in large transformers are not well behaved. The LLM.int8() authors found that once models reach about 6.7 billion parameters, a few hidden dimensions carry values up to 20 times larger than the rest, in every layer. At that scale, about 150,000 such outliers occurred per sequence, concentrated in just 6 feature dimensions, and zeroing them badly damaged the model. One per-token scale for activations would be dominated by these few values, crushing everything else towards zero: the shop with one expensive item in the aisle.

Three families of fixes:

• Keep outliers in high precision. LLM.int8() splits each multiply: the few outlier dimensions run in 16-bit, while more than 99.9 percent of the values are multiplied in 8-bit.
• Move the difficulty into the weights. SmoothQuant divides each activation channel by a factor and multiplies the matching weights by the same factor, a mathematically equivalent change that tames the activations while the weights, which are easy to quantize, absorb the range.
• Use a format with more range. FP8 copes better with a few large values than INT8.

Smarter rounding for low-bit weights

Round-to-nearest is the baseline. At 4 bits and below, methods that use a small calibration set do much better:

• GPTQ quantizes weights one column at a time and adjusts the not-yet-quantized weights to compensate for each rounding error, using approximate second-order information from calibration data.
• AWQ observes that a small share of weights matter most, the ones that meet large activations, and scales those channels up before quantizing so they lose less precision.

Both are post-training methods: no retraining, minutes to hours of compute. The alternative, quantization-aware training, simulates rounding during training so the model learns to tolerate it, at the cost of a training run.

Quality impact and how to check it

Typical results are encouraging. 8-bit weights, and 8-bit weights and activations with outlier handling, are usually close to the 16-bit model. Good 4-bit weight methods are usually close too, and a study across model families found 4-bit precision almost always the best use of a fixed memory budget: a bigger model in 4-bit tends to beat a smaller one in 8-bit. Below 4 bits, quality drops faster.

But averages hide the tasks that suffer. Long reasoning chains, maths, code and less common languages can degrade more than a headline score suggests. Treat a quantized model like any new model version: run your own eval set ( Building an eval set ) and compare it with the 16-bit model on the metrics that matter for your product before switching traffic.

Try it

1. Start at 4-bit with one scale for all 64. Note the error and how many weights became zero. Now switch to 8-bit. Why does the error fall so much?
2. Back at 4-bit, turn on the outlier. Count the weights rounded to zero. Then switch to groups of 16 and of 8. Which groups still suffer, and what does each choice cost in bits per weight?
3. Try 3-bit and 2-bit with groups of 8. How far do the stored values drift from the originals, and would you trust this model without evals?

The same effects on a real matrix multiply, with weights in several formats and an activation vector with one outlier channel:

# Quantize one layer y = W x and measure how far y moves from the 16-bit answer.
import numpy as np

rng = np.random.default_rng(0)
W = rng.normal(0, 0.02, (512, 512))       # weights: well behaved
x = rng.normal(0, 1.0, 512)               # activations for one token
x_out = x.copy(); x_out[7] = 20.0         # same, plus one channel 20x larger

def quant(a, bits, group):
    """Symmetric round-to-nearest, one scale per `group` values along the last axis."""
    L = 2 ** (bits - 1) - 1
    g = a.reshape(*a.shape[:-1], -1, group)
    scale = np.abs(g).max(axis=-1, keepdims=True) / L
    return (np.clip(np.round(g / scale), -L, L) * scale).reshape(a.shape)

def err(y, ref):
    return np.linalg.norm(y - ref) / np.linalg.norm(ref)

print("weight-only (activations stay 16-bit)")
for bits, group in [(8, 512), (4, 512), (4, 128), (4, 32), (3, 128)]:
    Wq = quant(W, bits, group)
    print(f"  W{bits}, {group:>3} weights per scale: error {err(Wq @ x, W @ x):6.2%}")

print("\nweights and activations in 8-bit (one scale per token)")
for name, a in [("normal token ", x), ("outlier token", x_out)]:
    y = quant(W, 8, 512) @ quant(a, 8, 512)
    print(f"  {name}: error {err(y, W @ a):6.2%}")

# LLM.int8()-style fix: keep the outlier channel in 16-bit, the rest in 8-bit
keep = np.abs(x_out) > 6
xa = np.where(keep, 0, x_out)
y = quant(W, 8, 512) @ quant(xa, 8, 512) + W[:, keep] @ x_out[keep]
print(f"  outlier kept in 16-bit: error {err(y, W @ x_out):6.2%}")

weight-only (activations stay 16-bit)
  W8, 512 weights per scale: error  0.72%
  W4, 512 weights per scale: error 14.76%
  W4, 128 weights per scale: error 12.21%
  W4,  32 weights per scale: error 10.17%
  W3, 128 weights per scale: error 25.58%

weights and activations in 8-bit (one scale per token)
  normal token : error  0.96%
  outlier token: error  3.18%
  outlier kept in 16-bit: error  0.73%

8-bit weights barely move the output. 4-bit weights with round-to-nearest move it by 10 to 15 percent of its size, and smaller groups help, which is why real 4-bit methods add groups and calibration on top. With 8-bit activations, one outlier channel more than triples the error, and keeping just that channel in 16-bit, the LLM.int8() trick, brings it back down.

Going deeper

Why decode speedups fall short of the bit ratio

Going from 16 to 4 bits cuts weight bytes by about 3.9 times once scales are counted, but measured decode speedups are usually smaller. The kernel must unpack and rescale every weight, which costs instructions; the KV cache is still read at its own precision; and at large batch sizes the step becomes compute-bound, where weight-only quantization does not help. Quantizing the KV cache to 8 bits ( GPU memory math ) attacks the second problem; W8A8 or FP8 attacks the third.

Choosing for your workload

• Memory-bound, small batches, latency-sensitive (chat with few concurrent users, local models): weight-only 4-bit or 8-bit.
• High-throughput serving with large batches and long prompts: FP8 or INT8 weights and activations, on hardware that has fast 8-bit units.
• Model does not fit at all: 4-bit weights may be the difference between one GPU and several, and the freed memory goes to the KV cache.

Quantized formats also matter for fine-tuning: QLoRA trains small adapters on top of a frozen 4-bit base model ( Parameter-efficient fine-tuning: LoRA and QLoRA ).

Trade-offs

Common mistakes

• Judging a quantized model on one average benchmark score and missing the tasks it got worse at.
• Expecting 4-bit weights to speed up prefill or large-batch serving, which are compute-bound.
• Forgetting the scales when computing memory: "4-bit" with small groups is 4.5 bits or more per weight.
• Quantizing activations with one scale per tensor and being surprised by outlier damage.
• Choosing a format the hardware cannot run natively, so the kernel spends its time converting numbers.

In the wild

Check yourself

Recap