Quantization

Quantization is a model compression technique that reduces the numerical precision of a neural network's weights and activations—for example, converting 32-bit floats to 8-bit or 4-bit integers—to shrink model size and speed up inference. It can be applied after training (post-training quantization, PTQ) or during training (quantization-aware training, QAT). The core tradeoff is between memory/compute savings and a small, often recoverable, drop in model accuracy.

Deploying large language models and vision models at scale demands hardware efficiency that full-precision weights cannot provide; quantization is one of the primary tools companies use to fit models onto GPUs, edge devices, and consumer hardware without retraining from scratch. Techniques like GPTQ, AWQ, and QLoRA have made it routine to run 70B-parameter models on a single consumer GPU, making quantization expertise directly tied to cost reduction and product viability. AI teams in 2026 list quantization alongside pruning and distillation as core skills for any ML engineer working on model serving or on-device inference.