Quantization: What Actually Breaks at 4 Bits

Imagine measuring everyone in an office with a tape marked in millimetres, then throwing that tape away and giving everyone a new one marked only in whole inches. You have lost detail. But if all you need is "who fits through this doorway," nobody notices.

AI models are full of numbers — billions of them — and each is normally stored with a lot of decimal places. Quantization means storing them much more coarsely. Fewer decimal places, less space.

The stakes are practical. A large model at normal precision needs about 140 gigabytes just to hold its numbers, which means several data-centre graphics cards. Stored four times more coarsely it needs about 35, which fits on one. That is the difference between a model you can run and a model you can read about.

Squashing every number to a quarter of its size sounds like it should be a disaster. In practice a well-quantized model is close to indistinguishable from the original on most tasks. The interesting question is not why does this work — it is where exactly does it stop working, because the failure is neither gradual nor evenly spread.

1. What is actually happening

Take a group of numbers and map them onto a small set of whole numbers. The standard recipe:

$$w_q = \mathrm{round}\!\left(\frac{w}{s}\right) + z$$

where $s$ is a scale — how much one step is worth — and $z$ shifts things so the range does not have to be centred on zero. To use the number you reverse it: $w \approx s(w_q - z)$.

With 4 bits you get 16 distinct values. Sixteen rungs on a ladder. If a group of numbers runs from $-0.8$ to $+0.9$, those 16 rungs get spread across that span, and every number snaps to the nearest rung.

And here is the consequence that drives every design decision after it: the extremes set the spacing.

One rogue number at $+8.0$, in a group where everything else is under $0.5$, stretches the ladder to cover eight units. Now all the ordinary numbers are crushed onto one or two rungs and are, effectively, all the same number.

A single outlier can destroy the precision of everything sharing its ladder.

2. Use more ladders

The fix is to not share so widely.

One ladder per whole grid is cheapest and maximally vulnerable to the problem above.

One ladder per row is much better, because outliers tend to cluster in particular rows and now only damage their own.

One ladder per group of 64 or 128 neighbouring numbers is what modern methods use, and that group size is the number you see in filenames like g128.

Fine-grained ladders cost something: each group stores its own scale and offset, so 4-bit weights actually cost around 4.25 bits once you count the bookkeeping. That overhead buys back most of the quality, which is a good trade and why nobody uses one ladder per grid any more.

3. The harder half: the numbers that appear at runtime

The model's stored numbers are known in advance, examinable offline, and never change. But there is a second set — the values flowing through the model as it processes your text. Those are computed live, depend on what you asked, and contain far more extreme outliers.

Researchers found something specific here. In large models, a small number of positions develop systematically enormous values — orders of magnitude above everything else. They appear consistently past a certain model size, they show up in the same positions across different inputs, and they matter: zeroing them badly damages the model.

That combination is why naive coarse storage of the runtime values fails on large models while working fine on small ones. And it is why the field settled on weight-only quantization: store the model's own numbers coarsely, keep the runtime values precise, and unpack the stored numbers as you use them.

That sounds wasteful. It is not, for the reason that makes this whole area make sense.

4. Quantization is really about moving bytes

Writing one word from a large model reads every number in the model out of memory and does relatively little arithmetic with each. The chip is waiting on memory, not on calculation — the same diagnosis behind FlashAttention and speculative decoding.

So 4-bit numbers are fast for a reason that has nothing to do with integer arithmetic being cheap: there are four times fewer bytes to fetch. The unpacking is essentially free, because it happens to data the chip was already waiting for.

This explains an otherwise puzzling observation. Quantization speeds up one-person-at-a-time use substantially, and does much less for serving many people at once — because when the same numbers serve many requests, the chip is no longer waiting on memory and the bottleneck has moved.

It also explains why keeping the runtime values precise is not a compromise. They are small compared to the model, and they were never the bottleneck.

5. The methods, and what separates them

Just round everything is the baseline: snap each number to its nearest rung, independently. Free, instant, and noticeably lossy at 4 bits.

GPTQ (Frantar and colleagues) treats it as a problem to solve rather than a thing to do. Quantize the numbers one column at a time, and after each one, adjust the columns you have not done yet to compensate for the error you just introduced. The model is not merely rounded — it is corrected, so the layer's output stays as close as possible to the original. This needs a small sample of real text to work against, and is meaningfully better than plain rounding.

AWQ (Lin and colleagues) starts from a different observation: not all numbers matter equally. Around 1% are important, and you can identify them not by their own size but by the size of the values they get multiplied against. Rather than keeping those in high precision — which breaks the neat hardware layout — AWQ scales them up before quantizing so they land on the ladder more precisely, and compensates elsewhere. Because it keys off runtime behaviour rather than reconstructing the original numbers, it tends to hold up better on text unlike its sample.

GGUF and the k-quants are the formats you meet running models on your own machine. The naming encodes the recipe: Q4_K_M means 4-bit, k-quant, medium — in practice most parts at 4 bits with the sensitive ones kept larger.

6. Where it actually breaks

The failures are not evenly spread, and knowing the pattern is the practical payoff.

Long answers degrade before short ones. The error per word is small and it compounds across a long chain of predictions. A model that answers factual questions perfectly can drift noticeably over two thousand words of writing.

Rare knowledge goes first. Facts repeated constantly in training are stored redundantly and survive. Obscure facts live in fine distinctions that 16 rungs cannot preserve. The standard quality score barely moves while the model quietly gets worse at exactly the specialist knowledge you might have chosen it for.

Step-by-step reasoning is fragile. Multi-step arithmetic or logic needs many consecutive correct steps, so a small error rate per step multiplies out. This is the most commonly reported real-world degradation.

Small models suffer much more. A 70-billion model at 4 bits usually beats a 13-billion model at full precision. A 1-billion model at 4 bits can be genuinely broken. The redundancy is what absorbs the error, and small models have less of it. So the general rule — prefer a bigger model more coarsely stored — holds until the model gets small, and then reverses.

Below 4 bits it falls off a cliff. 3-bit is usable with care. 2-bit generally is not, for anything that matters.

7. Choosing, in practice

If the model fits at 8 bits, use 8 bits. The degradation is close to unmeasurable and you skip the entire question.

If it does not, 4-bit with a good method — GPTQ or AWQ, group size 128 — is the default and genuinely fine for most work. Lean toward AWQ if your subject is far from ordinary prose.

Then test it on your own task. The standard quality score everybody reports is the least informative number available: it is an average over ordinary text, and every failure above is concentrated in the unusual cases, the long answers, and the multi-step reasoning. A quantized model with an identical score can be materially worse at the thing you need.

Thirty of your own test cases will tell you more than any benchmark table.

8. The framing

Quantization is not compression in the ordinary sense. It is a deliberate reduction in the resolution of a system that turns out not to need most of its resolution — and it pays out in data movement, which was the scarce resource all along.

Which is the same story as FlashAttention and speculative decoding, arrived at from a third direction. The consistent lesson of making AI models run faster is that the constraint is almost never arithmetic. It is moving bytes, and every technique that has stuck is a way of moving fewer of them.

Sources

• Frantar, Ashkboos, Hoefler, Alistarh — GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers, 2022
• Lin, Tang, Tang, Yang, Dang, Han — AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration, 2023
• Dettmers, Lewis, Belkada, Zettlemoyer — LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale, 2022