GGUF quantization explained — why Q4_K_M, and when it is the wrong choice

If you have downloaded an open model to run locally, you have picked a file with a name like Llama-3.2-3B-Instruct-Q4_K_M.gguf and probably not known what half of that meant.

Here is the whole thing, without the linear-algebra detour.

What quantization does

A model's weights are trained as 16-bit floating point numbers. A 3B model at 16 bits is roughly 6 GB — too big for any phone.

Quantization stores those same weights with fewer bits. Q4 uses about four bits per weight instead of sixteen, so the file drops to roughly a quarter of its size. That is the entire idea. You are not removing parameters; you are storing each one more coarsely.

The cost is precision. Each weight lands on the nearest available value rather than its exact one, and those tiny errors accumulate through the layers. Push it far enough and the model gets measurably worse: weaker at reasoning first, then at instruction-following, then it starts producing nonsense.

Reading the name

Take Q4_K_M apart:

• Q4 — about 4 bits per weight.
• _K — "K-quant". Rather than one scaling factor for a large block of weights, K-quants store weights in small super-blocks each with its own scale and minimum. Far more accurate than the old Q4_0 at almost the same size. If you see a quant without _K, it is the older scheme.
• _M — medium. Within K-quants there are S / M / L variants that differ in which tensors get spent extra bits. _M keeps the attention and feed-forward tensors that matter most at higher precision and squeezes the rest.

So Q4_K_M reads as: four-bit K-quant, medium mix, spend the extra bits where they buy the most quality.

Why nearly everything on a phone is Q4_K_M

Because of where the curve bends.

Going from 16-bit to Q8 costs you almost nothing in quality and halves the size. Going Q8 to Q4_K_M halves it again and costs a little. Going below Q4 — Q3, Q2 — starts costing a lot, fast, and small models suffer worst because they have less redundancy to spare.

Q4_K_M sits right at the knee: about a quarter of the original size, with quality loss that most people cannot detect in normal chat use. On a device where you get roughly 44% of physical RAM for weights, that ratio is what makes a 3B model possible on a 6 GB phone at all.

Rough guide at the same parameter count:

The exception: when Q4 is the wrong call

Every model in our catalogue is Q4_K_M except one. Gemma 3 270M ships Q8_0.

The reasoning is arithmetic. At 270M parameters, going from Q8 to Q4 saves about 37 MB. On a phone where the smallest budget is 817 MB of weights, 37 MB buys you nothing — there is no device where the Q4 version fits and the Q8 version does not.

Meanwhile the quality cost of Q4 is worst at this size. A 270M model has very little redundancy; the same proportional error that a 7B model absorbs, a 270M model shows. Google and Unsloth both recommend Q8 at this size for exactly that reason.

So the rule is not "always Q4_K_M". The rule is: quantize until the size actually unlocks a device, then stop. Below about 1B parameters, Q4 usually stops unlocking anything and starts only costing quality.

This is the check worth doing yourself. If dropping a quant level does not move the model into a tier your phone can hold, do not drop it.

What quantization does not fix

Two things people expect it to help with, and it does not:

Speed is mostly memory bandwidth, not size. A smaller file does load faster, and fewer bits per weight does mean less data moved per token, so Q4 genuinely is quicker than Q8. But the gain is far smaller than the size ratio suggests, because the bottleneck on Apple silicon is how fast weights stream from memory into the GPU, and dequantization costs compute on the way. Do not expect four times the tokens per second from a quarter of the file.

Context memory is unaffected. The KV cache is separate from the weights and it is not quantized by default. A long conversation costs the same RAM whether the model is Q8 or Q4. This is why a model that loads fine can still run out of memory forty messages later.

The short version

• Q4 = ~4 bits per weight. _K = the better block scheme, always prefer it. _M = the sensible medium mix.
• Q4_K_M is the default for a reason — it is the knee of the quality-per-byte curve.
• Below ~1B parameters, use Q8_0. The Q4 saving is too small to unlock a device and the quality cost is at its worst.
• Quantize to cross a hardware threshold. If it does not cross one, do not do it.