Reference · Glossary
Quantization
Compressing model weights to **fewer bits** (e.g. 16-bit → 8-bit or 4-bit) so models run faster and use less memory with small accuracy trade-offs.
When to use
Running LLMs on laptops, edge devices, or cutting inference cost at scale.
When not to
When you need maximum quality on hard reasoning benchmarks and have GPU budget to spare.
Example
A 7B model at 4-bit quantization may fit in 6 GB VRAM instead of 14 GB — slightly softer on nuance, much cheaper to serve.