NVIDIA-Nemotron-3.5-Bolt-E4B-OptiQ-4bit

mlx-community/NVIDIA-Nemotron-3.5-Bolt-E4B-OptiQ-4bit#

Built with mlx-optiq, the MLX-native toolkit to quantize, fine-tune, and serve LLMs locally on Apple Silicon, no PyTorch and no cloud. Try the Lab · All OptiQ quants · Docs

NVIDIA's Nemotron 3.5 Bolt, quantized for a Mac. A mixed-precision MLX quant of nvidia/NVIDIA-Nemotron-3.5-Bolt-E4B-BF16, built on the Gemma-4 E4B architecture. 7.5 GB on disk, down from 16 GB bf16. See the original card for what the model is trained to do and how NVIDIA evaluates it.

OptiQ quantizes the language tower to mixed 4/8-bit and keeps the vision tower at bf16 in a sidecar, so one checkpoint does both text and images.

What it is#

Following llama.cpp's naming for mixed quants, the "4bit" label denotes the predominant precision, not the weighted average.

How the bit-widths were chosen#

The per-layer allocation was not measured on this model. It was transferred from mlx-community/gemma-4-e4b-it-OptiQ-4bit, whose allocation came from a KL-divergence sensitivity sweep over a six-domain calibration mix. The two language towers share an architecture exactly, and all 345 tensors matched with none unmatched.

Checked after conversion#

Run through this quant and its bf16 parent on Apple Silicon with greedy decoding and thinking off:

Image input: shown a red rectangle, a blue circle and the number 42, it answered "The image displays a red square, a blue circle, and the number 42 below them on a white background."

Serve it with the assistant drafter#

Google's Gemma-4 E4B assistant drafter shares this model's tokenizer and architecture, and speeds it up roughly 2x. Measured on a 24 GB M4, batch 1, greedy, thinking off, best of three runs each:

pip install mlx-optiq
optiq serve --model mlx-community/NVIDIA-Nemotron-3.5-Bolt-E4B-OptiQ-4bit \
            --drafter mlx-community/gemma-4-e4b-it-assistant-bf16

The drafter's guesses were accepted 74% of the time on code and 45% on prose, even though it was trained for the base Gemma rather than this fine-tune. Every token is still verified by the model itself: code, math and the edit came out identical to plain decoding; the prose run took a different word at one position where the model's top two choices were within bf16 rounding of each other. For edits that copy the prompt back, --ngram-draft 16 is faster still.

Use it#

pip install mlx-optiq
# chat, images, and every app that talks to optiq serve
optiq serve --model mlx-community/NVIDIA-Nemotron-3.5-Bolt-E4B-OptiQ-4bit
optiq lab --model mlx-community/NVIDIA-Nemotron-3.5-Bolt-E4B-OptiQ-4bit

Text only, with stock mlx-lm:

from mlx_lm import load, generate

model, tokenizer = load("mlx-community/NVIDIA-Nemotron-3.5-Bolt-E4B-OptiQ-4bit")
messages = [{"role": "user", "content": "What is 17 multiplied by 23?"}]
prompt = tokenizer.apply_chat_template(messages, add_generation_prompt=True, tokenize=False)
print(generate(model, tokenizer, prompt=prompt, max_tokens=400))

Thinking is on by default under mlx-lm and optiq serve: the model writes a thought channel before the answer. Pass enable_thinking=False to apply_chat_template to turn it off; through optiq serve, send "reasoning_effort": "none" or "chat_template_kwargs": {"enable_thinking": false}. NVIDIA recommends temperature 1.0 and top-p 0.95 in both modes.

See the Gemma-4 family guide for family-specific notes.