GGUF quantization guide

I like running my own LLMs locally. Open models are becoming more and more powerful, with exciting releases like the latest Qwen 3.5 family scoring highly in benchmarks even in their smaller variants. This makes managing and running your own models more viable, as it becomes increasingly easy to repurpose old hardware for local inference with progressively better results. For local users and modest purposes, the GGUF format introduced by llama.cpp is the de-facto default.

Since local inference is typically heavily restricted by the available hardware, several optimization techniques have been implemented to make the models leaner and faster. Perhaps the most important of these is quantization, which trims down the bit count per parameter to achieve lower memory usage and (sometimes) faster inference. The challenge is that there are many different formats and strategies for quantization. In this post, I summarize them, providing a bird’s-eye view on the available techniques, their strengths, and their weaknesses.

Naming conventions

Most GGUF quantization names follow this pattern: Q{bits}{method}{size}

Unquantized formats

These are base formats with no compression. Usually, but not always, models are trained in these formats.

These preserve maximum accuracy but require significant VRAM. Most local users quantize to reduce size by 75-90%.

Legacy quantization formats (Q*_0, Q*_1)

Simple per-block linear quantization. Fast but less accurate at low bits.

Following is a table that contains the format name, number of bits, size (for a 7B model), perplexity, and some notes/recommendations. The perplexity represents the difference between the quantized model and the base model, with lower scores indicating better accuracy and less uncertainty.

K-quants or I-quants are generally preferred over legacy formats at \(\leq 5\) bits. Q8_0 remains useful for compatibility.

K-Quant formats (modern default)

Use two-level block quantization (small blocks \(\rightarrow\) super-blocks) with double-quantized scales for better quality-per-bit.

Q4_K_M seems to be the community sweet spot, as it offers ~75% size reduction with minimal noticeable quality loss for most tasks.

I-Quant formats (aggressive compression)

IQs use non-linear reconstruction, lookup tables, and importance-matrix calibration for maximum quality at very low bit counts. They trade decoding speed for size.

I-quants require importance matrix (imatrix) calibration during quantization for best results. Without it, quality can degrade noticeably.

Special formats

Decision guide

• Best balance \(\rightarrow\) Q4_K_M (default recommendation)
• Maximum quality, still compressed \(\rightarrow\) Q5_K_M or Q6_K
• Tight VRAM, acceptable quality loss \(\rightarrow\) Q4_K_S or IQ4_XS
• Absolute smallest file \(\rightarrow\) IQ3_XS / Q3_K_S (test quality first)
• Near-original accuracy \(\rightarrow\) Q8_0 or keep FP16/BF16
• Running on CPU \(\rightarrow\) Q4_K_M or IQ4_NL for better decode speed
• Fitting a 70B model on 24GB VRAM \(\rightarrow\) IQ3_S or Q3_K_M + CPU offload

Conclusions

As you see, this looks like the wild west at first glance, but this mess is not without order. Most importantly, there are good reasons behind every quantization type. Here are some takeaways.

1. Bits \(\neq\) quality alone: A well-designed 4-bit format (Q4_K_M, IQ4_XS) can outperform a naive 5-bit legacy format.
2. Suffixes matter: _M variants selectively keep sensitive layers at higher precision, improving quality with minimal size cost.
3. Hardware matters: I-quants compress more but decode slower on CPUs; K-quants often give better tokens/sec on consumer hardware.
4. Calibration helps: Models quantized with an importance matrix (imatrix) retain more accuracy, especially at \(\leq 3\) bits.
5. Test your use case: Perplexity benchmarks are only guides. Always validate outputs for your specific tasks.

For the latest format support and benchmarks, check the llama.cpp repository or community hubs like Hugging Face, where curators like bartowski and Unsloth publish tested GGUF variants.