What is model quantization and what are the tradeoffs of running an LLM in INT8 or INT4?

Quantization stores model weights (and sometimes activations) at lower numeric precision, converting FP16/BF16 to INT8, INT4, or FP8. Since LLM inference is largely memory-bandwidth bound, this roughly halves or quarters memory and speeds up loading and generation, letting large models fit on smaller GPUs. Post-training quantization (GPTQ, AWQ, bitsandbytes) needs no retraining; quantization-aware training bakes robustness in during training. The tradeoff is accuracy: aggressive INT4 can degrade quality, especially on reasoning, and outlier activations are hard to represent, so methods keep sensitive channels in higher precision or use per-group scales. INT8 is usually near-lossless; 4-bit is a popular quality/size sweet spot for local deployment. KV caches can also be quantized to save memory during long-context generation.