Qwen3.8-27B-GPTQ-4bit

mlx-community/Qwen3.8-27B-GPTQ-4bit#

Quantized by sady4850. Previously sady4850/Qwen3.8-27B-4bit-GPTQ, the old id redirects here.

Qwen/Qwen3.8-27B in exactly the layout of mlx-community/Qwen3.8-27B-4bit (MLX affine 4-bit, group size 64), with the round-to-nearest codes, scales and biases of every quantized language-model Linear replaced by GPTQ.

Same tensors, shapes, dtypes and config as the template, so it is a drop-in replacement wherever the template loads (mlx-vlm, mlx-lm, Splash) with the same size (15 GB) and speed. Only the numbers inside the 4-bit tensors differ.

• 497 Linears requantized (all 64 layers incl. lm_head). The embedding and the vision tower are the template's.
• GPTQ fitted to MLX's own affine format: each group's scale and bias are computed from the error-updated weights and rounded to bf16 before coding, so the result dequantizes exactly as written.
• Hessians from the bf16 model, one per shared input, damp 0.01, block 128.
• Calibration: 119 x 2048 tokens. 64 chunks of UltraChat train conversations (chat template applied) and 55 chunks of calibration_v5 text.

Use#

pip install -U mlx-vlm
python -m mlx_vlm.generate --model mlx-community/Qwen3.8-27B-GPTQ-4bit --max-tokens 100 --prompt "Hello"

In Splash use this repo id in place of mlx-community/Qwen3.8-27B-4bit.

Quality vs bf16#

Texts: wiki = wikitext-2 test, chat = first 300 conversations of UltraChat test as plain text. 30 chunks x 2048 tokens per text, scored on the second half of each chunk (like llama-perplexity). KLD = KL(bf16 || quant) of the next-token distribution, flips = top-1 token differs from bf16.

Note that chat is close to the calibration data (UltraChat train vs test split), wiki is out of domain.

torch (A100), both checkpoints dequantized into the same bf16 model:

Splash (teacher-logits, int8 KV, M1 Pro), the engine's own logits against bf16 log-probs for the same rows (every 8th scored row, KL over the bf16 top-512 tokens plus a bound for the rest, within 2% of full-vocab KL):

Speed in Splash is the same as the template: median per 2048-token window 14.10 s prefill for both, 23.00 s vs 23.05 s for 127 verify cycles.

PPL alone is a poor measure of closeness here: the RTN checkpoint has a lower chat PPL than bf16 itself. Lower KLD and fewer flips mean the outputs follow the original model more closely.

Agent replay#

Independent run by @ghisguth (splash discussion #34): 78 recorded turns of a real coding-agent session (32K to 124K context), each turn sent with the same recorded prompt, temperature 0, Splash 1.4.0-m1 on M2 Max, compared per turn against Q8_0.

Same calls = same tools with the same arguments as Q8_0. For scale, Q8_0 against itself gives 82%. Speed was the same as RTN (cycles/s within 2%). 78 turns is a small sample, but it agrees with the KLD above.

Other quants in Splash#

Same windows and bf16 reference as above, for choosing between formats:

If the extra 1.5 GB and ~11% slower decode (short context, less at long context) are fine, UD-Q4_K_M is closer to bf16. At the size and speed of MLX 4-bit, this repo is the closer one.

Limitations#

Vision was not evaluated (all runs are language-only), calibration is English chat and text, KLD windows are 2048 tokens. Task accuracy was not measured.

Raw numbers and logs in eval/.