Calibrated 4-bit MLX quantization of poolside/Laguna-S-2.1 (118B total, 8B activated per token), produced with oMLX oQ at level 4 enhanced — 4.60 bits/weight effective, 64 GB on disk. Data-driven mixed precision: bits are allocated per tensor from an imatrix-calibrated sensitivity map, not a fixed rule. For Apple Silicon.
64 GB on disk, down from 235 GB BF16
48 layers, 47 of them MoE with 256 routed experts + 1 shared, top-10 (L0 is a dense MLP); interleaved attention (12 global with YaRN to 1M context, 36 sliding-window 512)
Peak memory in my tests: 60.4 GB at 1k context, 63.5 GB at 64k — fits a 96 GB Mac
Converted and tested on a Macbook Pro M5 Max 128GB 40 GPU
mlx-lm doesn't support the laguna architecture yet — there's an open PR: mlx-lm#1223. Until it lands, use mlx-vlm (0.6.3+), which implements laguna as a text-only model:
oMLX serves it directly from 0.5.3 on — it vendors that PR and patches it into mlx-lm at import, so no model setting is needed. On earlier builds, discovery decides between the mlx-lm and mlx-vlm loaders by looking for a vision sub-config, and laguna has none — so it lands on mlx-lm and fails with Model type laguna not supported. Set model_type_override: "vlm" in the model's settings, then refresh discovery (omlx restart): the load failure is cached per entry until the next discovery pass, so setting the override alone won't clear it.
oQ4e allocates bits per tensor from an importance-matrix calibration pass over calibration data. The 4-bit base lands on the experts; the dense spine — attention, embeddings, lm_head, routers, 386 tensors in total — came out mixed, 284 at 8 bits, 1 at 6 and 101 at 5. Output is standard MLX affine quantization — no custom kernels or runtime required.
oQ at level 4 enhanced — imatrix-calibrated, group size 128. I had to patch omlx to route laguna through the mlx-vlm loader. At 235 GB the model doesn't fit in 128 GB of RAM, so calibration ran against a uniform 4-bit proxy on disk rather than the FP weights, which shifts the bit allocation slightly.
Smoke-tested after conversion with mlx_vlm.generate: coherent — solved 17 * 24 = 408, broke it down by the distributive property and verified the result a second way, no repetition loop.
mmlu_pro, mathqa and winogrande, n=300 seeded samples each, thinking off, identical questions across every variant. The bf16 row is the hosted API, measured the same way. Standard error at this n is around 2.5 points, so oQ4e through oQ6e aren't separated by this run.
Treat this as a rough sighting, not a verdict. Three benchmarks at n=300 cover a narrow slice of what the model does — no long-context work, no agentic loops, no real code — and at this sample size most of the ladder above 3.6 bpw sits inside the error bars. I ran them to size the drop between levels, not to rank the variants against each other. Test the one you're considering on your own workload before trusting any of it.
# mlx-vlm — plain mlx-lm doesn't support the laguna architecture
uvx --from mlx-vlm mlx_vlm.generate --model mlx-community/Laguna-S-2.1-oQ4e \
--prompt "Explain Bayes' theorem in two sentences." --max-tokens 300
# oMLX — discovers the model from the HF cache; set model_type_override: "vlm" first
omlx serve
import os
os.environ["HF_ENDPOINT"] = "https://freeweights.allternit.com"
from transformers import AutoModelForCausalLM, AutoTokenizer
tok = AutoTokenizer.from_pretrained("mlx-community/Laguna-S-2.1-oQ4e", revision="d785a9349850807a34ac0ac1c22c66b718e77881")
model = AutoModelForCausalLM.from_pretrained("mlx-community/Laguna-S-2.1-oQ4e", revision="d785a9349850807a34ac0ac1c22c66b718e77881", device_map="auto")
mlx-community <huggingface.co/mlx-community>Newsgroups: alt.freeweights.textSubject: 
