Quantization: Making Big Models Fit Small Computers
Why Q4, Q8, GGUF, and suspiciously tiny model files keep appearing in local AI guides.
Quantization shrinks a model by storing its numbers with less precision. The model becomes smaller and easier to run, usually at some quality cost.
This is why local model names contain codes like Q4, Q5, Q8, GGUF, and other strings that look like firmware versions for a device you regret buying.
The tradeoff
Higher precision usually means better quality and more memory use.
Lower precision usually means smaller files, faster local inference, and more chance the model gets worse at subtle tasks.
Q4 is often a practical starting point for local use. Q8 is larger and closer to the original behavior. The exact best choice depends on model, task, and hardware.
GGUF
GGUF is a common file format used by llama.cpp and many local runners. If you are downloading a local model for Ollama, LM Studio, or llama.cpp-based tooling, you will see it often.
The practical rule
Pick the largest quantized model your machine can run comfortably, then test it on your actual tasks. If it is slow, go smaller. If it is dumb, go larger or use a hosted model.