Last updated on August 17, 2026

Quantization

weightsFP8INT4memory

Reduces the numerical precision of weights and activations (for example, FP16 to FP8 or 4-bit) to shrink memory footprint and sometimes speed up inference.

The short answer

Pick a format that your hardware and serving engine support efficiently, then choose the lowest bit width that passes evaluation on your workload. On Hopper or Blackwell GPUs, a validated FP8 checkpoint is often a strong production default because tensor cores accelerate FP8. On Ampere or older hardware, 4-bit weight-only AWQ or GPTQ can make a model fit, with quality and speed that depend on the model and kernel. On a laptop or Apple Silicon, GGUF through llama.cpp or Ollama is the usual path.

Quantization changes how many bytes each weight occupies. That is the entire mechanism: at BF16 a weight costs 2 bytes, at FP8 or INT8 it costs 1, at 4-bit it costs 0.5.
Memory is the reason to do it. Speed can also improve through lower memory traffic or native low-precision arithmetic, but the gain depends on kernels, hardware, batch size, and model shape.
W8A8 formats such as FP8 and INT8 SmoothQuant quantize activations too, so they help buy both memory and real arithmetic speedup on supported hardware.
Below roughly 4 bits per weight, quality degrades sharply. 2-bit and 3-bit are for research and for cases where the alternative is not running the model at all.

At a glance

Also called: Low-bit inference; post-training quantization (PTQ)
Optimizes for: Memory footprint, then throughput
Typical saving: 2x at 8-bit, 4x at 4-bit
Quality cost: Depends on the model, method, calibration, and evaluation task
Applies to: Weights, activations, and KV cache
Where it runs: Usually offline; dynamic and load-time paths also exist

Theoretical memory reduction from BF16 to 4-bit weights

140 to 35 GB

Weight footprint of a 70B model, BF16 versus 4-bit

2 GPUs to 1

80GB H100s needed to hold a 70B model, BF16 versus 4-bit

0.5 bytes

Cost of a single weight at 4-bit precision

Calculator

Will it fit?

Pick a model, a GPU, and a precision. The calculator works out the weight footprint, what is left for KV cache, and roughly how much context you can serve. All arithmetic is shown so you can check it.

Model Hardware Precision

Fits. 16.0 GB of weights on 24.0 GB, leaving about 4.0 GB for KV cache.

Weights: 16.0 GB
Usable VRAM: 20.0 GB
Left for KV cache: 4.0 GB
Memory-limited context, batch 1: 31K tokens

Show the arithmetic

weights   = 8B params × 2 bytes = 16.0 GB
budget    = 24.0 GB ÷ 1.2 overhead = 20.0 GB
kv/token  = 2 × 32 layers × 8 kv heads × 128 head dim × 2 bytes = 131,072 bytes
kv budget = 20.0 − 16.0 = 4.0 GB
context   = 4.0 GB ÷ 131,072 bytes = 30,517 tokens

Figures use decimal GB (1 GB = 10^9 bytes) and assume a 20% overhead allowance for activations, CUDA workspace, and allocator fragmentation. Real engines vary. Treat the result as a go / no-go signal, not a guarantee.

Which format should you use?

Read this top to bottom and stop at the first row that matches your hardware.

Format	Bits (W/A)	Best on	Use it when
FP8	8 / 8	Ada, Hopper, Blackwell	You have a supported engine, native FP8 hardware, and a validated checkpoint. Can reduce memory and accelerate matrix multiplication with little quality loss on tested workloads.
NVFP4 / MXFP4	4 / 4	Blackwell	You are on B200 or newer and want both the memory and the compute win. Newest option, least battle-tested.
INT8 (SmoothQuant)	8 / 8	Hardware with a supported INT8 kernel	You want W8A8 inference on hardware and an engine where a validated SmoothQuant path is available.
AWQ	4 / 16	GPUs with a supported AWQ kernel	You need the model to fit and have a well-made AWQ checkpoint supported by your runtime.
GPTQ	4 / 16	GPUs with a supported GPTQ kernel	Same use case as AWQ. Pick whichever has a good published checkpoint for your model.
GGUF (k-quants)	Mixed / runtime-dependent	CPU, Apple Silicon, mixed	You are running locally through llama.cpp or Ollama. Q4_K_M is a common starting point; benchmark lower or higher encodings for your workload.
bitsandbytes NF4	4 / 16	Supported CUDA GPUs	You want zero-setup on-the-fly quantization for experiments or QLoRA. Not the fastest choice for serving.

Engine support matrix

Support moves fast and varies by model architecture. Verify against the docs for the version you are running before you commit.

Format	vLLM	SGLang	TensorRT-LLM	llama.cpp	LMDeploy	MLC-LLM
BF16 / FP16	Yes	Yes	Yes	Yes	Yes	Yes
FP8 (W8A8)	Yes	Yes	Yes	No	Yes	Partial
INT8 (SmoothQuant)	Yes	Yes	Partial	No	Yes	No
AWQ (W4A16)	Yes	Yes	Yes	No	Yes	Partial
GPTQ (W4A16)	Yes	Yes	Yes	No	Yes	No
GGUF k-quants	Yes	Partial	No	Yes	No	No
NVFP4 / MXFP4	Yes	Yes	Yes	Partial	Partial	No
bitsandbytes NF4	Yes	Partial	No	No	No	No

Verified August 2026.

Compare all inference engines →

Quality-loss ledger

What each original paper actually claims. Numbers are as published by the authors, not re-measured here.

Method	Precision	What the source reports	Year
LLM.int8()	W8A8	No degradation up to 175B parameters, using mixed-precision decomposition to keep outlier feature dimensions in 16-bit.	2022
SmoothQuant	W8A8	Preserves accuracy on OPT-175B by migrating quantization difficulty from activations into weights via a per-channel scale.	2022
GPTQ	3-4 bit weight-only	Quantizes OPT-175B in roughly 4 GPU hours with negligible perplexity increase at 4-bit. Degradation grows at 3-bit.	2022
AWQ	4-bit weight-only	Activation statistics identify salient weights; an equivalent per-channel scaling transformation reduces quantization error without mixed precision, backprop, or reconstruction.	2023
QLoRA (NF4)	4-bit weight-only	Introduces 4-bit NormalFloat, which is information-theoretically optimal for normally distributed weights, plus double quantization of scale constants.	2023
FP8 formats	W8A8	E4M3 and E5M2 match 16-bit training and inference results across a range of model classes without changing hyperparameters.	2022

What precision do current open models actually ship in?

These values describe recorded release precisions rather than every available third-party conversion. Verify the source model card before deployment.

Model	Parameters	Shipped precision	Recommended hardware
DeepSeek-V4-Pro-0813	1.6T (49B active)	FP4 + FP8 Mixed	8× H200, 8× B200
Qwen3.8-2.4T-A95B	2.4T (95B active)	BF16FP8	8× B300 (NVFP4), 8× MI355X (MXFP4)
Kimi-K3	2.8T (104B active)	MXFP4 weights + MXFP8 activations	8× B300, 8× MI355X
DeepSeek-V4-Flash-0731	284B (13B active)	FP4 + FP8 Mixed	8× H200, 8× MI325X
DeepSeek-V4-Pro	1.6T (49B active)	FP4 + FP8 Mixed	8× H200, 8× B200

How it works

What quantization actually does

A trained model is a large pile of numbers. Quantization stores those numbers in fewer bits.

That is the whole idea. A weight held in BF16 occupies 2 bytes. Move it to FP8 or INT8 and it occupies 1. Move it to 4-bit and it occupies half a byte. Nothing about the architecture changes, no layers are removed, and the parameter count stays exactly the same. Only the container gets smaller.

The reason this works is that trained neural networks often tolerate rounding many weights to a smaller set of representable levels. Weight distributions commonly cluster around zero. Group-wise scaling lets a quantizer use the smaller set of levels efficiently while storing scale factors alongside the packed values.

The difficulty is preserving the important weight and activation values that do not fit the simplified numerical range well.

The memory math

Two numbers determine whether a model fits. Both are simple arithmetic, and neither requires running anything.

Weights. Multiply the parameter count by the bytes per parameter.

weight_bytes = parameters × bytes_per_parameter

BF16  = 2 bytes    FP8/INT8 = 1 byte    raw 4-bit value = 0.5 byte

A 70 billion parameter model at BF16 needs 70 × 2 = 140 GB of weights. The same model at 4-bit needs 70 × 0.5 = 35 GB. That single line is why a 70B model that requires two H100 GPUs at BF16 fits on one at 4-bit.

KV cache. This is the number people forget, and on long-context workloads it dominates.

kv_bytes_per_token = 2 × layers × kv_heads × head_dim × bytes_per_element
total_kv_bytes     = kv_bytes_per_token × sequence_length × batch_size

The leading 2 accounts for storing both a key and a value. For Llama 3.1 70B at FP16, that is 2 × 80 × 8 × 128 × 2 = 327,680 bytes per token, or roughly 0.33 MB. At 128K context, a single sequence needs about 43 GB of KV cache, which is larger than the entire 4-bit weight footprint of the model. Quantizing weights does nothing to that number, so if long context is your bottleneck, start with what is KV cache and KV cache offloading instead.

Then add overhead. Activations, runtime workspace, and allocator fragmentation vary by engine and workload. The calculator uses a conservative working rule:

total ≈ (weights + kv_cache) × 1.2

Everything above uses decimal GB, where 1 GB is 10^9 bytes. GPU labels and runtime tools do not always present capacity with the same unit convention: tools often report MiB or GiB even when product names use GB (If you convert to GiB you will get numbers about 7% smaller).

Why outliers are the hard part

Naive 8-bit activation quantization can work on smaller models and lose accuracy at larger scales. LLM.int8() reported that, in the transformer families studied, systematic outlier feature dimensions emerged at about 6.7B parameters and grew more prominent with scale.

Quantization works by mapping a range of values onto a fixed set of levels. One value a hundred times larger than the rest stretches that range so far that every ordinary value collapses onto the same few levels, destroying the information they carried.

Several influential W8A8 methods address this problem directly:

LLM.int8() isolates the outlier dimensions and computes them in 16-bit, quantizing only the well-behaved remainder.
SmoothQuant divides activations by a per-channel scale and multiplies the corresponding weights by it, mathematically shifting the difficulty from activations, which are hard to quantize, into weights, which are easy.
AWQ uses activation statistics to identify salient weights, then applies an equivalent per-channel scaling transformation that reduces weight quantization error without mixed precision.
FP8 sidesteps a good deal of the problem structurally, because keeping an exponent field gives the format a much wider dynamic range than INT8 at the same bit count.

If you remember one thing about W8A8 research since 2022, remember that activation outliers are a major part of the challenge. Weight rounding error, calibration, scaling granularity, and kernel implementation still matter.

Practical defaults

If you want a decision without reading the tables:

Production serving on H100, H200, or B200: start with a validated FP8 checkpoint when the model and engine support one, then confirm quality and throughput on the target workload.
Production serving on A100: evaluate INT8 SmoothQuant for W8A8 throughput or AWQ when weight memory is the primary constraint.
Single consumer GPU: start with a well-supported AWQ or GPTQ 4-bit checkpoint; group size 128 is common, but kernel and checkpoint quality matter more than a universal group-size rule.
Laptop, CPU, or Apple Silicon: a GGUF file through llama.cpp or Ollama, with Q4_K_M as a common starting point.

Whichever you pick, the format has to be supported by the runtime you serve with, so check the inference engines page before committing to a checkpoint. For the precision each current open model ships in, and the hardware the authors recommend for it, see the model pages.

Trade-offs

Memory always improves. Speed sometimes does.

Weight-only 4-bit formats store packed low-bit weights and usually dequantize them inside a fused kernel for FP16 or BF16 computation. You reduce weight storage and memory traffic, but you do not get native W4A4 arithmetic. On a compute-bound workload, weight-only 4-bit can be no faster than BF16 and can be slower when dequantization and kernel overhead dominate.

Small batches gain more than large ones.

Decoding at low batch size is often memory-bandwidth-bound, so reading fewer weight bytes can reduce token latency substantially. As batch size rises, weight reuse increases and the workload can become compute-bound, reducing the benefit of weight-only quantization. The crossover depends on the model, hardware, and kernel.

The KV cache is a separate budget.

Quantizing weights does nothing for the KV cache, which grows linearly with cached sequence length and concurrent sequences and can exceed the weight footprint on long-context workloads. KV cache quantization is a separate engine setting and should be evaluated independently for memory savings, hardware support, and quality.

Native hardware support is the dividing line.

FP8 on supported Ada, Hopper, and Blackwell GPUs and FP4 on Blackwell can use dedicated tensor-core paths. On hardware without native support, a runtime may reject the format, upcast it, or use a slower fallback. Check both the GPU generation and the serving kernel before choosing a format.

Frequently asked questions

Does quantization make LLM inference faster?

It depends on the format, kernel, hardware, and workload. Formats with native tensor-core support can accelerate matrix multiplication. Weight-only 4-bit formats such as AWQ and GPTQ mainly reduce memory traffic, which often helps most at low batch size where decoding is bandwidth-bound; the benefit can shrink as your workload becomes compute-bound.

How much VRAM does a 70B model need?

Roughly 140 GB for the weights at BF16, about 70 GB at FP8 or INT8, and about 35 GB at 4-bit. Add 15 to 25% on top for activations, CUDA workspace, and allocator fragmentation, then add the KV cache, which depends on your context length and batch size.

AWQ or GPTQ, which is better?

Both are weight-only methods with similar footprints, but quality and speed depend on the model, checkpoint recipe, group size, kernel, and runtime. AWQ avoids backpropagation and reconstruction, while GPTQ uses approximate second-order information. The practical answer is usually to benchmark well-made checkpoints for your specific model and serving engine.

Is FP8 better than INT8?

Not universally. FP8 keeps an exponent field and provides wider dynamic range than INT8, which can simplify activation quantization on supported hardware. INT8 with a method such as SmoothQuant can still preserve accuracy and perform well. Hardware support, scaling strategy, kernels, and model evaluation should decide between them.

Does quantization shrink the KV cache?

Weight quantization alone does not. Weight and KV cache quantization are separate things. The KV cache grows with cached sequence length and concurrent sequences and is unchanged by the stored weight format. Many inference engines expose a separate low-precision KV cache option. Validate quality and kernel support before enabling it.

Should I quantize a model myself or download a quantized checkpoint?

Choose a checkpoint released by the model authors, because they know which layers to hold at higher precision. Failing that, quantize it yourself with calibration data drawn from your actual workload, which is generally better than a community checkpoint calibrated on generic web text. Community re-quantizations are the least predictable option because the recipe is usually undocumented.

Sources

Keep exploring

Optimization