Qwen3.5-9B-OptiQ-4bit

mlx-community/Qwen3.5-9B-OptiQ-4bit#

Built with mlx-optiq, the MLX-native toolkit to quantize, fine-tune, and serve LLMs locally on Apple Silicon, no PyTorch and no cloud. Try the Lab · All OptiQ quants · Docs

A 4-bit mixed-precision MLX quant produced by mlx-optiq, the sensitivity-aware quantization toolkit for Apple Silicon. Beats stock uniform 4-bit on every benchmark in the six-metric Capability Score.

A 4-bit mixed-precision MLX quant of Qwen/Qwen3.5-9B. Per-layer bit-widths come from a KL-divergence sensitivity pass on a six-domain calibration mix (prose · reasoning · code · agent · tool-call · constraint-bearing instructions). Sensitive layers go to 8-bit; robust ones stay at 4-bit. The on-disk size is within ~5 % of a stock uniform 4-bit MLX quant.

Quantization details#

We follow the same naming convention llama.cpp uses for Q4_K_M and similar mixed-precision quants: the "4-bit" label is for the predominant precision, not the weighted average. The mixed allocation is what lets this build beat stock uniform-4-bit on every benchmark below at the same disk size.

Usage#

Load it with mlx-lm and use it as usual:

pip install mlx-lm

from mlx_lm import load, generate

model, tokenizer = load("mlx-community/Qwen3.5-9B-OptiQ-4bit")
response = generate(
    model, tokenizer,
    prompt="Explain quantum computing in simple terms.",
    max_tokens=200,
)

For more (mixed-precision KV-cache serving, sensitivity-aware LoRA fine-tuning, OpenAI + Anthropic-compatible inference server, hot-swap mounted adapters, sandboxed Python execution for agent workflows), install mlx-optiq:

pip install mlx-optiq

Speculative decoding (MTP)#

This quant ships with a bundled Multi-Token Prediction head as mtp.safetensors. Enable it for ~1.4× faster decode:

optiq serve --model mlx-community/Qwen3.5-9B-OptiQ-4bit --mtp

Acceptance rate stays ~70% at depth 2 (the empirical sweet spot for Qwen3.5).

Using the head outside optiq serve: optiq/mtp.safetensors holds Qwen's original MTP tensors, so its seven RMSNorm weights (mtp.*norm*) are zero-centred, applied as 1 + w. The main weights have the +1 folded in, as MLX expects. Add 1.0 to all seven norms when loading the head into another runtime.

See the Qwen3.5 family guide on mlx-optiq.com for sampling defaults, training recipes, and family-specific caveats.

Benchmarks#

Six-metric Capability Score (mean of MMLU + GSM8K + IFEval + BFCL + HumanEval + HashHop). Apples-to-apples comparison against stock uniform 4-bit:

Every metric gets one equal vote. Disk size is reported next to the score as an honest second axis instead of being folded into the score. See the eval-framework writeup for the full methodology.

Links#

• Project website: mlx-optiq.com
• Qwen3.5 family guide: mlx-optiq.com/docs/qwen3.5
• PyPI: pypi.org/project/mlx-optiq
• Calibration mix: mlx-optiq.com/blog/calibration-mix
• Eval framework: mlx-optiq.com/blog/eval-framework
• Base model: Qwen/Qwen3.5-9B

Quantize your own#

This quant was produced by mlx-optiq. Point it at any Hugging Face model to get the same sensitivity-aware mixed precision:

pip install mlx-optiq
optiq convert <hf-model-id> --target-bpw 5.0 --candidate-bits 4,8
optiq lab   # full local workbench: chat, compare, quantize, fine-tune

License#

Apache 2.0 (inherits from base model).