Quantization Is Not a Zip File: Model Layers and the Reality of 4-bit Inference

LLM quantization is often reduced to one sentence: turn 16-bit weights into 4-bit weights and the model becomes four times smaller. That is true, but it omits nearly every difficult part.

Quantization first solves a capacity problem: the weights do not fit in VRAM or unified memory. It also offers a possible second benefit—higher speed. Decode is frequently memory-bandwidth-bound, and the weights must be read again for every token. Yet a smaller file does not automatically run faster. If the runtime expands 4-bit weights to F16/F32, or the device lacks a packed kernel, much of the capacity and bandwidth advantage disappears.

Quantization is therefore not a file-conversion option. It is an execution path spanning model specification, weight assembly, device residency, kernels, and quality validation.

1. Start with layers: model architecture and weight encoding are different concerns

zLLM does not put everything behind one opaque Model object:

model_spec/<model>  architecture constants: layers, hidden size, attention, MoE
runtime/<model>     LayerSpec expansion and complete prefill/decode orchestration
weight/model        checkpoint names, shape validation, model assembly
weight/format       byte layouts such as FP8, W4A16, and GGUF K-quants
weight/codec        IO-free reference decoding and layout conversion
backend + kernel    residency, format dispatch, and direct device computation

The important consequence is that model architecture is not weight encoding. The same linear layer may come from BF16, FP8, NVFP4, W4A16, or Q4_K. The format layer interprets bytes, the model adapter knows names and shapes, runtime describes dataflow, and the backend selects a kernel according to device capability.

2. What quantization actually solves

Capacity comes first

Ignoring scales, metadata, and tensors retained at higher precision, an N-parameter model needs approximately:

BF16 / FP16: 2N bytes
INT8:        1N bytes
INT4:        0.5N bytes

A 70B model is therefore about 140 GB in BF16 and ideally 35 GB at 4-bit. Real Q4_K_M files are larger because block scales, minima, alignment, and protected tensors also consume space. Quantization only shrinks the data it covers: KV cache, activations, scratch buffers, and runtime memory still need separate budgets.

On discrete GPUs this determines whether weights fit in VRAM. On Apple UMA it determines whether the model, KV cache, OS, and application can coexist without severe memory pressure. Eliminating a traditional host-to-VRAM staging copy does not make capacity infinite.

Moving fewer bytes can make decode faster

Single-token decode is dominated by GEMV-like operations with little weight reuse. If weights stay packed and a kernel decodes each local block directly into accumulation, 4-bit storage offers close to four times the effective weight density of BF16.

That speedup requires a native device kernel, low enough decode/scale/activation overhead, and end-to-end measurement. In one real zLLM Qwen3.8 CPU path, packed Q4_K AVX2/FMA dot products reduced a 16-token decode from 107.483 to 7.578 seconds, and paired gate/up traversal reached 7.183 seconds. This roughly 15× change combined removal of repeated F32 expansion with specialized vectorization; it is not a universal 4-bit-versus-BF16 ratio. Conversely, a Gemma 4 Q4_K dual-output kernel passed its CPU oracle but reduced end-to-end throughput from 35.9 to 35.5 tok/s and was reverted.

3. The core idea: local structure, not global rounding

Naive quantization shares one range across an entire matrix:

q = round(w / scale)
w_hat = scale * q

Outliers stretch that range and leave too few useful grid points for ordinary values. Modern weight quantization therefore makes three local decisions.

Block locality. Groups of 32, 64, 128, or 256 weights receive their own scale. Smaller groups fit local distributions better but add metadata and kernel cost.

Tensor and channel sensitivity. Attention value/output projections, FFN down projections, embeddings, and the LM head need not tolerate the same error. AWQ similarly uses activation statistics to identify and protect salient weights.

Layer locality. Error propagates through depth: a hidden-state shift changes later routing, attention, and token ranking. Layerwise calibration controls accumulated error better than one global conversion. MoE calibration must also exercise a representative expert-routing distribution.

How to read common GGUF quantization names

GGUF is a container, not a quantization algorithm. It stores metadata, tokenizer data, a tensor directory, and tensors that may independently use F32, F16, Q4_K, Q6_K, IQ4_XS, and other types. A Q4_K_M filename does not mean every tensor uses an identical 4-bit block.

Family	Common names	Structure and use	Practical guidance
Classic blocks	Q4_0, Q5_0, Q8_0	Integer codes plus a block scale; simple and widely implemented	Q4_0 favors compatibility; Q8_0 is a useful quality baseline or activation format
K-quants	Q3_K, Q4_K, Q5_K, Q6_K	Sub-block scales/minima inside 256-weight super-blocks	Start with Q4_K_M; use Q5_K_M/Q6_K when memory permits
I-quants	IQ2_, IQ3_, IQ4_NL, IQ4_XS	Importance-aware or nonlinear codebooks	Strong quality per bit, but verify native backend kernels
Mixed recipes	Q3_K_S/M/L, Q4_K_S/M, Q5_K_S/M, UD-*	Different tensor types according to sensitivity	Read the tensor directory; never infer support from the filename alone

S/M/L usually means Small, Medium, or Large mixed-precision policy, not three integer widths. IQ4_XS and Q4_K_M also are not interchangeable: the former uses nonlinear/codebook ideas, while the latter uses hierarchical K-quant scales. A custom name such as UD-Q4_K_XL may mix Q4_K, Q5_K, Q6_K, and IQ4_XS across layers.

4. Why 4-bit is often the sweet spot

The sweet spot is where three curves meet: capacity falls sharply, devices can still decode efficiently, and quality has not entered the steep decline often seen at 2–3 bits.

For Mistral 7B on WikiText-2 at context 512, one same-method llama.cpp comparison reported:

Format	Perplexity	Increase over FP16
Q3_K_S	6.0021	5.44%
Q3_K_M	5.8489	2.75%
Q4_K_S	5.7349	0.75%
Q4_K_M	5.7259	0.59%
Q5_K_S	5.7100	0.31%

Moving from Q3_K_M to Q4_K_M cuts the relative increase from 2.75% to 0.59%; moving again to Q5_K_S only reaches 0.31% while consuming more memory bandwidth. Llama 33B results show a similar curve: FP16, Q6_K, Q5_K_M, Q4_K_M, and Q3_K_M scored 4.1557, 4.1598, 4.1675, 4.2081, and 4.3594 respectively.

These numbers establish a trend only for that model, dataset, context, and quantizer. Perplexity is not synonymous with chat quality, code correctness, or long-reasoning stability. Small local errors can flip tokens near a decision boundary or change MoE Top-K routing.

5. Why models can be quantized at all

Neural networks are not bit-exact symbolic programs. Many parameters form redundant statistical representations, while residual connections, normalization, and overparameterization tolerate small perturbations. Quantization exploits that redundancy.

Generation is probabilistic as well. Logits pass through softmax, temperature, top-k/top-p, and sampling. At nonzero temperature, identical weights can produce different answers; even greedy decoding can flip when two logits are extremely close.

This does not mean errors are harmless. Open-ended generation often distinguishes good from better rather than one exact string, so quality must be evaluated statistically and by task. Mathematics, code, tool arguments, and JSON schemas still have hard correctness boundaries.

A quantized model may occasionally score above the floating-point baseline. Quantization noise may break an existing bad preference, but finite samples, sampling variance, and evaluator noise can do the same. Only repeated, multi-seed, multi-dataset results with uncertainty justify a claim of genuine improvement.

6. What training and calibration compensate for

Post-training quantization (PTQ) uses representative samples to estimate scales, clipping, channel importance, or Hessian approximations. GPTQ compensates layerwise for induced weight error; AWQ uses activations to protect salient weights. The target is not merely “each weight remains close,” but “the layer output remains close on real inputs.”

Quantization-aware training (QAT) simulates rounding and clipping during training so parameters adapt to the low-precision grid. It costs more but becomes increasingly important for 3/2-bit weights, activation quantization, and sensitive tasks. Calibration data must match deployment: short prose is a poor calibration set for long reasoning, code, or broad MoE expert routing.

Weight, activation, and KV-cache quantization are separate choices. W4A16 keeps activations at 16-bit; W4A8 adds dynamic-range and fused-kernel constraints; KV quantization changes long-context capacity and continuously perturbs attention. Calling all three a “4-bit model” hides essential information.

7. Fences constrain structure; they do not manufacture correctness

Three engineering fences are needed:

1. Operator fence: reference codecs, CPU oracles, layerwise hidden/logit comparisons, and real shape validation.
2. Generation fence: token-level JSON Schema, tool-name, closing-tag, stop-token constraints, plus an independent repetition guard.
3. Quality fence: greedy regression prompts, perplexity, task suites, long contexts, multi-seed evaluation, and human review for critical tasks.

A generation fence can guarantee allowed structure, not correct semantics. A JSON repair pass must not hide tool-selection drift. Chain-of-thought should not be judged only by verbatim equality either: visible reasoning is sampling-sensitive and is not the model's internal computation. Evaluate final answers and tool outcomes, key intermediate constraints, finish reasons, loop rate, length distribution, and success-rate changes under the same decoding policy.

Short greedy prompts can produce a dangerous false pass. Small logit shifts accumulate over thousands of reasoning tokens. Long tests should track correctness, semantic restarts, repetition, premature EOS, and pathological long tails. Fences prevent protocol disasters; evaluations discover semantic regression.

8. Engineering pitfalls

“Support” must be split into four states:

file can be parsed
≠ CPU reference can decode it
≠ device has a packed direct kernel
≠ model uses it in the default end-to-end path

Packed data must remain packed until the kernel. Expanding it to F32 at load time erases the memory and bandwidth advantage. The desired lifecycle is container lookup → format interpretation → name/shape validation → backend residency → block-local decode and accumulation.

Do not forget non-weight memory. KV cache, prefill activations, MoE working sets, vision encoders, logits, and command buffers can become the new peak. Measure TTFT, prefill and decode throughput, peak memory, SSD wait, GPU utilization, and thermal steady state. Finally, “the output looks fluent” is not correctness: validate codecs, operator oracles, layerwise differences, fixed-token regressions, and full task suites.

Conclusion

Quantization reallocates a budget among precision, capacity, bandwidth, and compute. Four bits is often attractive because it crosses the capacity threshold, retains near-baseline statistical quality, and maps well to packed device kernels. It is not a magic number independent of model, data, hardware, and task.

A trustworthy deployment answers four questions: what is quantized at what granularity; whether the device consumes the packed representation directly; whether speed improves end to end; and whether quality fences cover long reasoning, structured output, and real tasks. Only then does “the model runs” become “the model is deployable.”

References

• GPTQ
• AWQ
• llama.cpp Mistral 7B quantization comparison
• llama.cpp K-quant perplexity records
• llama.cpp quantization formats