Quantization
Shrinking a model (for example to 4-bit numbers) so it runs on smaller hardware, with a small quality loss.
Last updated
Analogy
Compressing images to WebP: much smaller, and most people can't tell.
In an agent
Lets you run open models locally or on cheap servers for narrow agent tasks.
Practice
Flashcards
- What is quantization?Shrinking a model so it runs on smaller hardware, with a small quality loss.
Questions people ask
What is model quantization?
Quantisation shrinks a model by storing its numbers with fewer bits, for example 4-bit instead of 16-bit. The model needs much less memory and runs on smaller hardware, with a small loss in quality. It works like compressing images to WebP: much smaller, and most people cannot tell.
When should I use a quantized model?
When you want to run an open model on a laptop, a cheap server or limited GPU memory, especially for narrow tasks such as classification or simple extraction. Test it on your own examples first: for hard reasoning, the quality loss can matter more than the memory saved.