flint-oQ2.7e

flint-oQ2.7e#

Recursive Flint (40B total, ~2.5B active per token) quantized for Apple Silicon with oMLX's oQ2.7e: data-driven mixed precision with imatrix-calibrated clipping.

• 16.2 GB on disk (~3.26 bpw), down from 79.6 GB bf16. Peak memory while generating: ~16.3 GB at short context, ~17.8 GB at 16k.
• MoE: 49 layers (2 dense + 47 MoE), 256 routed experts with top-6 + 1 shared expert. 35 sliding-window (513 tokens) + 14 full-attention layers, 262k context.
• Loads in oMLX and mlx-vlm ≥ 0.7.10 (see Runtime).
• Quantized and tested on a MacBook Pro M5 Max 128GB 40 GPU.

For 24 GB Macs. Heavy loss against bf16 (perplexity +8): it still answers coherently in my probes, but expect more mistakes. No faster than oQ4e.

Quantization#

oQ level 2.7, enhanced (oQ2.7e), group size 64, affine. The allocator measured each layer's sensitivity on the bf16 weights and puts the routed experts at 2 bits, with every down_proj and the gate_proj/up_proj of the 11 most sensitive MoE layers raised to 3 bits. Everything read on every token stays at 8 bits: attention, the shared expert, the dense layers, embed_tokens and lm_head. The "e" adds an imatrix pass (128 calibration samples) that picks the clipping range of each group by activation importance; it doesn't change which tensor gets how many bits, only the scales inside them.

On oQ4e I also tried group size 128 for every tensor. It saved 1.2 GB but lost everything the imatrix bought (mean KL 0.263 vs 0.222) and wasn't faster, so every variant uses 64.

Built with oMLX e2396c7 (2026-10-27). Output is standard MLX affine quantization: no custom kernels needed to load it.

Fidelity vs bf16#

How closely each variant tracks the bf16 model's next-token distribution, measured over 64 × 2048-token chunks (65,472 scored positions, second half of each chunk), KL truncated to the bf16 top-256.

The corpus is English prose from oMLX's bench set, which isn't guaranteed disjoint from oQ's calibration data. That makes the numbers good for ranking these variants against each other, not as a held-out quality score. Truncating to top-256 drops ~0.9% of the probability mass and biases KL slightly low for every row alike. I haven't run task benchmarks yet.

Speed#

Generation tok/s, greedy decode, 256 tokens, median of 3 runs, oMLX's fused Flint decode path enabled, MacBook Pro M5 Max 128GB 40 GPU.

Each variant was timed against oQ4e in the same session; across sessions the absolute numbers drift by up to ~10%, so read differences under that as noise. Below oQ4e, fewer bits buy memory but not speed: at ~2.5B active parameters, decode is mostly bound by kernel-launch overhead. Above it, oQ5e, oQ6e and oQ8e are 5–15% slower.

Runtime#

oQ quantizes Flint through mlx-vlm's model class, so the weights are stored under a language_model. prefix. oMLX and mlx-vlm load them as-is; mlx-lm's own flint model expects unprefixed names.

oMLX ships a fused MoE routing/combine kernel for one-token Flint decode, worth ~16% generation speed in my runs on oQ4 (~108 → ~125 tok/s at short context). It's on by default in oMLX builds from 2026-10-27 onward.

Usage#

oMLX (recommended): download the repo and it shows up as mlx-community--flint-oQ2.7e. The server exposes an OpenAI-compatible API with tool calls and returns reasoning in reasoning_content.

hf download mlx-community/flint-oQ2.7e
omlx serve

mlx-vlm:

pip install -U "mlx-vlm>=0.7.10"
mlx_vlm.generate --model mlx-community/flint-oQ2.7e \
  --prompt "Write a Python function that merges overlapping intervals." \
  --max-tokens 4096 --temperature 1.0

Recursive recommends temperature 1.0, top-p 0.95, top-k 20, and an output budget of at least 32k tokens for hard problems, since reasoning and answer share it. Keep the returned reasoning in the conversation history across turns. The chat template is the upstream one, unchanged.

See the original model card for training, evaluations, and limitations.

License#

Apache 2.0, same as the original weights. Flint is developed by Recursive; this repo only changes the weight precision. Recursive's hosted API terms are separate from the weight license.