Reference · Glossary
Quantization
Compressing model weights to **fewer bits** (e.g. 16-bit → 8-bit or 4-bit) so models run faster and use less memory with small accuracy trade-offs.
#When to use
Running LLMs on laptops, edge devices, or cutting inference cost at scale.
#When not to
When you need maximum quality on hard reasoning benchmarks and have GPU budget to spare.
#Example
A 7B model at 4-bit quantization may fit in 6 GB VRAM instead of 14 GB — slightly softer on nuance, much cheaper to serve.