gpt-oss-20b-OptiQ-4bit

mlx-community/gpt-oss-20b-OptiQ-4bit#

Built with mlx-optiq, the MLX-native toolkit to quantize, fine-tune, and serve LLMs locally on Apple Silicon, no PyTorch and no cloud. All OptiQ quants · Docs

A sensitivity-driven mixed-precision 4-bit MLX quant of OpenAI's gpt-oss-20b, produced by mlx-optiq. OptiQ measures each layer's quantization sensitivity and assigns per-layer bit-widths (4 or 8) under a target budget, instead of quantizing every layer to the same width.

gpt-oss ships natively in MXFP4 (its experts are already 4-bit). That makes a naive uniform-4-bit re-quantization unusually lossy, since it compounds error on top of weights that are already at the 4-bit floor. OptiQ keeps the layers that matter at 8-bit, so it recovers most of what uniform-4 drops.

Capability Score#

Six-benchmark mean (MMLU, GSM8K, IFEval, BFCL, HumanEval, HashHop), measured against a genuine uniform-4-bit baseline built from the same weights.

OptiQ wins all six. The largest gap is HashHop (long-context multi-hop retrieval), where uniform-4 collapses to 19% while OptiQ holds 78%: uniform 4-bit erodes the long-range attention precision that retrieval depends on, and OptiQ's mixed precision keeps those layers at 8-bit.

What it is#

Run it#

gpt-oss needs mlx-lm and import optiq (keep it for the rest of the toolkit):

pip install -U mlx-optiq "mlx-lm @ git+https://github.com/ml-explore/mlx-lm.git"

from mlx_lm import load, generate
from mlx_lm.sample_utils import make_sampler

model, tok = load("mlx-community/gpt-oss-20b-OptiQ-4bit")
prompt = tok.apply_chat_template(
    [{"role": "user", "content": "Explain mixed-precision quantization in one sentence."}],
    tokenize=False, add_generation_prompt=True, reasoning_effort="low",
)
print(generate(model, tok, prompt=prompt, max_tokens=512, sampler=make_sampler(temp=0.7)))

gpt-oss is a reasoning model using the harmony response format: it emits an analysis channel before a final channel. Pass reasoning_effort ("low", "medium", "high") to control how much it reasons, and read the answer from the final channel.

Serve it with the OptiQ Lab + OpenAI-compatible API:

optiq serve --model mlx-community/gpt-oss-20b-OptiQ-4bit

Notes#

• No KV-cache quantization. gpt-oss uses attention sinks, which mlx-lm's quantized-SDPA path does not support, so this quant ships without a kv_config.json. Weight quantization (the OptiQ result above) is unaffected.
• This is a weight-only quant; it loads under stock mlx-lm as a standard gpt_oss model.
• The full method is described at mlx-optiq.com/docs.