Model Formats Explained: GGUF vs GPTQ vs AWQ vs EXL2

📚 More on this topic: Quantization Explained · FP4 Inference: MXFP4 & NVFP4 on llama.cpp · llama.cpp vs Ollama vs vLLM · Text Generation WebUI Guide · VRAM Requirements

You’re on HuggingFace looking for a model. There are half a dozen uploads of the same thing: GGUF, GPTQ, AWQ, EXL3, an MLX build for Mac, maybe an FP4 (MXFP4/NVFP4) version, and plain SafeTensors. They’re all the same model at roughly the same size. Which one do you download?

The answer depends on your hardware and which inference tool you use. Each format is optimized for a different setup, and picking the wrong one means slower speeds or outright incompatibility. This guide breaks down what each format is, what runs it, and when to use it.

The Core Formats at a Glance

*Older Llama-2-13B-era figures — illustrative of relative ordering, not current absolute speeds.

These deliver nearly identical quality at 4-bit; the differences are speed, hardware compatibility, and tool support. Three newer entries sit alongside them in 2026: MLX (Apple Silicon’s native format), FP4 — MXFP4 and NVFP4 (4-bit floating-point, Blackwell-native, now in the GGUF ecosystem), and Unsloth’s UD- dynamic GGUF quants (the quality-leading GGUF variant). Each gets its own note below. The one-line summary that hasn’t changed: GGUF is the de-facto local format, and everything else is a specialization on top.

GGUF: The Universal Format

GGUF (GPT-Generated Unified Format) is the format used by llama.cpp. It’s the only format that runs on CPU, and it supports every GPU backend — CUDA, Metal, Vulkan, ROCm. If you use Ollama or LM Studio, you’re already using GGUF.

Why GGUF wins for most people:

• Runs everywhere: CPU, NVIDIA GPU, Apple Silicon, AMD GPU, Intel Arc, even Raspberry Pi
• Partial offloading: Split layers between GPU and CPU with -ngl. No other format does this.
• Single file: Everything the model needs — weights, tokenizer, architecture — in one portable file
• Widest quant range: From IQ1_S (1.69 bits/weight) to Q8_0 (8.5 bits/weight), plus the K-quant series that allocates more bits to important layers

Common GGUF quantization levels:

Start with Q4_K_M. It’s the community standard and the best balance of size, speed, and quality.

One upgrade worth knowing: Unsloth’s UD- “Dynamic 2.0” quants (and any imatrix/importance-matrix GGUF) push more bits into the layers that actually hurt when squeezed, so a UD-Q4_K_XL often reads closer to Q5 quality at Q4 size. If a model has a UD- or imatrix build on HuggingFace, prefer it over the plain quant at the same level.

→ Use our Planning Tool to check exact VRAM for your setup.

Tool support: Ollama, LM Studio, llama.cpp, KoboldCpp, text-generation-webui, GPT4All, Open WebUI

GPTQ: The Production GPU Format

GPTQ uses calibration data and second-order optimization to find the best quantized weights layer by layer. It’s GPU-only — the dequantization happens in CUDA kernels during inference.

Why use GPTQ:

• Fastest at scale with Marlin kernels: vLLM + GPTQ + Marlin hit 712 tok/s on Qwen 2.5 32B — 1.5x faster than FP16. This matters for serving multiple users.
• Widest GPU tool support: HuggingFace Transformers, vLLM, text-generation-webui, TensorRT-LLM, ExLlamaV2
• Mature ecosystem: Most models on HuggingFace have a GPTQ variant available

Why skip GPTQ:

• GPU-only. No CPU fallback, no partial offloading
• Without Marlin kernels, it’s actually slower than FP16 in vLLM (276 vs 461 tok/s in one benchmark)
• Slightly lower quality than AWQ and EXL2 at the same bitrate
• Requires CUDA 12.1+

Best for: Existing vLLM deployments and the many HuggingFace models that already ship a GPTQ variant. One caveat for 2026: the field has largely moved to AWQ as the default 4-bit GPU format (better quality retention, same throughput story on modern kernels), and GPTQ tooling — AutoGPTQ in particular — is fading. GPTQ still works and is everywhere, but if you’re starting fresh, reach for AWQ first.

Tool support: vLLM, HuggingFace Transformers, AutoGPTQ/GPTQModel, text-generation-webui, ExLlamaV2, TensorRT-LLM

AWQ: Best Quality at 4-Bit

AWQ (Activation-Aware Weight Quantization) takes a different approach. Instead of optimizing weights directly, it identifies the ~1% of weight channels that matter most based on activation patterns, then scales them up before quantizing. The result: better quality retention than GPTQ with no runtime overhead from the scaling.

Why use AWQ:

• Best quality at 4-bit: ~95% quality retention vs ~90% for GPTQ. Perplexity differences are small (4.33 vs 4.34 on Llama 2 13B) but AWQ consistently edges out GPTQ and holds up better on downstream tasks
• Better generalization: AWQ doesn’t overfit its calibration data, so it works well across different domains
• TensorRT-LLM native: If you’re using NVIDIA’s production stack, AWQ is a first-class citizen

Why skip AWQ:

• GPU-only, same as GPTQ
• Speed depends heavily on the inference engine — not always faster than GPTQ
• Fewer quantization options than GGUF (typically just 4-bit)

Best for: Most GPU serving in 2026. AWQ has quietly become the default 4-bit format for vLLM and TensorRT-LLM production stacks — quality preservation plus solid throughput on modern kernels. If you’re standing up a new multi-user deployment, this is the safe pick.

Tool support: AutoAWQ, vLLM, HuggingFace Transformers, text-generation-webui, TensorRT-LLM

EXL3 (and EXL2): Fastest Personal Inference

The ExLlama family’s signature trick is variable bits per weight — instead of quantizing every layer to the same bitrate, it measures which layers are sensitive and spends more bits there. You target any average bitrate (3.5, 4.65, 5.0, whatever fits your VRAM) rather than the nearest round number.

EXL3 has replaced EXL2 as the one to reach for. ExLlamaV3 shipped its stable release in mid-2026 (v1.1.0, July 2026) with trellis-based quantization that stays coherent down to ~1.6 bpw — genuinely usable low-bit quants, not just a party trick. New quants are landing as EXL3, and TabbyAPI/text-generation-webui support it. EXL2 still works and there’s a large back-catalog of EXL2 quants, but for a new download on NVIDIA, EXL3 is the better default.

Why use EXL3/EXL2:

• Fastest single-user inference on NVIDIA: roughly 1.5x faster than GGUF on the same hardware for a mid-size model, thanks to tight CUDA kernels
• Best quality at low bitrate: variable precision beats uniform 4-bit methods in perplexity, and EXL3’s trellis quantization extends that edge further down the bpw scale
• Arbitrary bitrate targets: squeeze a model into exactly the VRAM you have

Why skip it:

• NVIDIA only. No CPU, no Apple Silicon, no AMD
• Smaller ecosystem. ExLlamaV2/V3, TabbyAPI, and text-generation-webui — that’s basically it
• No Ollama/LM Studio support. If that’s your workflow, this isn’t an option

Best for: Power users on NVIDIA GPUs who want the fastest token generation and are comfortable with TabbyAPI or text-generation-webui. Especially good when you need to fit a model into a specific VRAM budget.

Tool support: ExLlamaV3/ExLlamaV2, TabbyAPI, text-generation-webui

Which Format Should You Use?

Follow this decision tree:

Do you use Ollama or LM Studio? → GGUF. No other format is supported.

Are you on CPU? → GGUF. It’s the only option.

Are you on Apple Silicon? → GGUF works and is the easy default. If you want a bit more speed and don’t mind a smaller toolset, MLX is Apple’s native format (see below).

Are you on AMD (ROCm)? → GGUF. The others have limited or no AMD support.

Are you on NVIDIA and want max personal speed? → EXL3 via ExLlamaV3 or TabbyAPI (EXL2 if that’s what the model ships).

Are you serving to multiple users? → AWQ via vLLM or TensorRT-LLM. (GPTQ + Marlin still works for existing setups.)

Do you need the best quality at 4-bit? → AWQ, or EXL3 at a higher bpw target. On a UD-/imatrix GGUF you’re already most of the way there.

Not sure? → GGUF Q4_K_M. It works everywhere, the quality is competitive, and you can always switch later.

MLX: Apple Silicon’s Native Format

If you’re on a Mac, GGUF is still the easy default — but MLX is worth knowing. It’s Apple’s own array framework, and MLX-format models run through the unified-memory architecture more directly than llama.cpp’s Metal backend does, so you often get a modest speed and memory-efficiency edge on M-series chips. LM Studio and the mlx-lm tools run it, and the mlx-community org on HuggingFace publishes MLX builds of most popular models.

The tradeoff is reach: MLX is Apple-only, the ecosystem is smaller than GGUF’s, and you lose the CPU/AMD/NVIDIA portability. Use it when you’re committed to a Mac and want to squeeze the hardware; stick with GGUF if you ever move models between machines.

FP4: The New Frontier (MXFP4 and NVFP4)

The newest entrants aren’t integer quants at all — they’re 4-bit floating-point. MXFP4 is an Open Compute Project microscaling standard (GPT-OSS ships in it natively), and NVFP4 is NVIDIA’s Blackwell-native variant with a second-level scale for better accuracy. Both keep a tiny exponent per weight, which holds up better than INT4 on the outlier-heavy activations that trip up older 4-bit methods — and Blackwell GPUs have hardware FP4 tensor cores to run them fast.

This isn’t just a datacenter story: FP4 has landed in the local ecosystem, including GGUF (GGML_TYPE_NVFP4), so you’ll start seeing FP4 builds on HuggingFace. Whether they beat a good Q4_K_M or AWQ on your hardware depends on the model and your GPU — this is early, and the honest answer is “measure it.” We ran the numbers on llama.cpp in a dedicated piece: FP4 Inference: MXFP4 & NVFP4 on llama.cpp.

What About Other Formats?

GGML: Deprecated in August 2023, replaced by GGUF. No modern tool supports it. If you find a GGML model, look for a GGUF version instead.

SafeTensors: Not a quantization format — it’s HuggingFace’s secure storage format for unquantized (FP16/BF16) model weights. SafeTensors files are the starting point from which all quantized versions are created. GPTQ, AWQ, and EXL2 models actually store their quantized weights inside SafeTensors files — only GGUF uses its own container.

BitsAndBytes (load_in_4bit/8bit): On-the-fly quantization in HuggingFace Transformers. Convenient but slow — 23 tok/s vs 31-64 tok/s for pre-quantized formats on the same hardware. Use it for quick testing, not production.

.bin (PyTorch pickle): Legacy model format. Being replaced by SafeTensors for security reasons (pickle files can execute arbitrary code when loaded).

Converting Between Formats

You generally don’t need to convert — HuggingFace has pre-quantized versions of popular models in every format. But if you need to:

FP16 SafeTensors → GGUF:

# Clone llama.cpp and use convert script
python convert_hf_to_gguf.py /path/to/model --outtype f16
./llama-quantize model-f16.gguf model-Q4_K_M.gguf Q4_K_M

FP16 SafeTensors → GPTQ:

pip install auto-gptq
# Use AutoGPTQ Python API with calibration dataset

FP16 SafeTensors → AWQ:

pip install autoawq
# Use AutoAWQ Python API

FP16 SafeTensors → EXL2:

# Use ExLlamaV2's convert.py
python convert.py -i /path/to/model -o /output -cf /output/4.65bpw -b 4.65

Most users never need to do this. The community provides pre-quantized versions in every major format — these days that means bartowski and mradermacher for broad GGUF coverage and Unsloth for the UD- dynamic quants. (TheBloke, who quantized half the models on HuggingFace in 2023–24, has gone quiet — if a page points you at a TheBloke repo, look for a fresher build from one of the above.)