LLM Quantization Types Explained 2026
Download pages list Q2_K through Q8_0, file sizes differ threefold, and nobody tells you what the quality gap actually is. Here is the cost of each level.
📜 Table of Contents
LLM Quantization Types Explained 2026: Q4, Q5, Q8 and What Each GGUF Level Costs You
Quantization stores weights in fewer bits: FP16 becomes 8-bit, 4-bit, even 2-bit. A 70B model needs well over 100GB at FP16 and roughly 40GB at 4-bit. The price is precision loss; the craft is putting that loss where it does not matter.
What the bits actually save
Rough formula: size ≈ parameters × bits per weight ÷ 8. An 8B model at FP16 is about 16GB theoretical, at 4-bit about 4GB. Real GGUF files run slightly larger because embeddings, output layers and quantization metadata often keep higher precision.
Level comparison
| Type | Approx bits/weight | Size vs FP16 | Quality expectation | Use when |
|---|---|---|---|---|
| Q2_K | ~2.5-3 | ~18-20% | Clearly degraded, long answers drift | Only if nothing else fits |
| Q3_K_M | ~3.5 | ~24-26% | Workable on big models, strained on small | Extreme compression |
| Q4_0 | 4 | ~28% | A notch below Q4_K_M | Legacy compatibility |
| Q4_K_M | ~4.5 | ~33% | The 2026 default sweet spot | First choice |
| Q5_K_M | ~5.5 | ~40% | Near-lossless on most tasks | Math, code, long reasoning |
| Q6_K | ~6.6 | ~47% | Very close to FP16 | Stability matters |
| Q8_0 | 8 | ~62% | Essentially lossless | Baselines and comparisons |
| FP16/BF16 | 16 | 100% | Reference | Training and eval baselines |
Percentages shift with architecture; only compare levels of the same model. File size on the download page is the most reliable signal.
Reading the filename
Qwen3-8B-Q4_K_M.gguf: Q4 means roughly 4 bits per weight; K means the improved k-quant block layout, better quality than old Q4_0 at similar size; _M means medium mix, while _S is more aggressive (smaller, worse) and _L closer to the original (bigger, better). Names containing imatrix or IQ used importance-matrix calibration and usually do better at equal size. Q4_K_M and Q4_K_S are not interchangeable.
How large the quality loss really is
The common finding: Q4_K_M loses very little against FP16 on general benchmarks, often inside measurement noise; Q5_K_M and Q6_K are closer still; Q2_K visibly degrades - repetition, broken reasoning chains, weaker instruction following.
There is one clear exception: math, code and multi-step reasoning are more sensitive. The same quantization that feels free in casual chat can produce obvious errors when solving equations or writing complex functions. Smaller models tolerate compression worse; a 3B model at Q3 is barely usable.
So ignore absolute claims that some level is lossless. Test with your own prompts.
Quantization buys memory, not necessarily speed
Generation is bound by memory bandwidth - every token streams the whole weight set. Four bits move half the bytes of eight bits, but dequantization costs compute, so on many machines Q4 and Q8 land much closer in speed than their sizes suggest. Quantization first buys "it fits", second "it runs fast".
Memory math for a 70B
FP16 weights ~140GB, Q4 ~40GB, Q8 ~87GB - before KV cache. Long contexts add several GB. Feasibility is about weights plus cache plus runtime overhead, not file size alone.
Four selection rules
- Default to Q4_K_M - best value for most work.
- Go up a level for math/code/long reasoning, or when the model is only 3B-7B.
- Only drop below Q4 if it will not fit - below Q3 you are usually better off with a smaller model.
- Use Q8_0 as your baseline when checking whether an effect comes from quantization.
Run your own A/B
Prepare 20 prompts you would actually send, covering simple Q&A, long summarization, code and multi-step reasoning. Run two levels of the same model with fixed temperature and seed. Check facts, formatting stability, and whether answers got shorter or lazier. Twenty minutes beats any leaderboard for your use case.
FAQ
Does quantization hurt Chinese? Yes, though usually later than English. Low-bit levels produce odd word choices; for Chinese-first work stay at Q4 or above.
IQ versus Q? IQ uses importance-matrix non-uniform quantization and is usually better at equal size, but needs newer loaders.
Why do two Q4_K_M files differ in size? Some quantize embeddings, some keep them at FP16, and some apply imatrix calibration. Read the release notes, not just the label.
Can I re-quantize back up? No. Quantization is lossy; upconverting only makes the file bigger.
Note: bit counts, size ratios and quality outcomes vary with quantizer version and architecture. Check official documentation and your own measurements.
Related
Written by our editorial team; tools listed here are tested or verified against public sources. Links point to official sites or GitHub repos for reference only — no paid placements.