Sources are tried in order. A file counts only when its size and sha256 match the manifest. Sources are checked when you ask for them, not on a timer.
Sources
Badge
Host
Last check
Result
From source
huggingface.co
2026-11-15 05:20 UTC
passed
Model card
From the publisher at commit cf20f37. Shown as plain text with links; images open at the publisher.
Note. At Mistral, we don't yet have too much experience with providing GGUF-quantized checkpoints to the community, but want to help improving the ecosystem going forward. If you encounter any problems with the provided checkpoints here, please open a discussion or pull request
Building upon Mistral Small 3.1 (2503), with added reasoning capabilities, undergoing SFT from Magistral Medium traces and RL on top, it's a small, efficient reasoning model with 24B parameters.
Magistral-Small-2506_gguf is a GGUF (quantized) version of Magistral-Small-2506.
Reasoning: Capable of long chains of reasoning traces before providing an answer.
lightweight: with its compact size of just 24 billion parameters, Magistral is light enough to run on a single RTX 4090 or a Mac with 32GB RAM once quantized, making it an appropriate model for local deployment and on-device use.
Apache 2.0 License: Open license allowing usage and modification for both commercial and non-commercial purposes.
Context Window: We recommend setting the context window to 40k. We converted the original weights to gguf with this requirement. Theoretically, an even larger 128k context window is supported but untested.
Tokenizer: Utilizes a Tekken tokenizer with a 131k vocabulary size.
You can interact with the model using the llama-cli's llama.ccp tool. Make sure to add --jinja to use our tokenizer. It uses the default system prompt for Magistral.
The default system prompt is in English, but you can customize it if you want by passing the -sys "your_system_prompt" argument to llama-cli.
By default, the context size of llama.ccp is 4096, but you can increase it to 40,960. We also recommend to set the temperature to 0.7 and the top_p to 0.95.
Now you can interact with the model directly where it is served in the browser, via curl or by using the OpenAI client.
from huggingface_hub import hf_hub_download
import openai
client = openai.OpenAI(
base_url="http://<your-url>:8080/v1",
api_key="not-needed",
)
def load_system_prompt(repo_id: str, filename: str) -> str:
file_path = hf_hub_download(repo_id=repo_id, filename=filename)
with open(file_path, "r") as file:
system_prompt = file.read()
return system_prompt
SYSTEM_PROMPT = load_system_prompt("mistralai/Magistral-Small-2506_gguf", "SYSTEM_PROMPT.txt")
completion = client.chat.completions.create(
model="Magistral-Small-2506_Q8_0.gguf",
messages=[
# The following line is not required if you use the default system prompt.
{"role": "system", "content": SYSTEM_PROMPT},
{"role": "user", "content": "How many 'r' are in strawberry?"}
],
temperature=0.7,
top_p=0.95,
stream=True
)
print("client: Start streaming chat completions...")
printed_content = False
for chunk in completion:
content = None
if hasattr(chunk.choices[0].delta, "content"):
content = chunk.choices[0].delta.content
if content is not None:
if not printed_content:
printed_content = True
print("\ncontent:", end="", flush=True)
# Extract and print the content
print(content, end="", flush=True)
Here is a list of questions to help you test the model.
How many "r" are in strawberry? The answer is 3.
John is one of 4 children. The first sister is 4 years old. Next year, the second sister will be twice as old as the first sister. The third sister is two years older than the second sister. The third sister is half the age of her older brother. How old is John? The answer is 22.
9.11 and 9.8, which is greater? The answer is 9.8.
Files
5 files · 72.2 GBWeights 25.1 GB (smallest of 2 GGUF variants). Sizes and sha256 from the signed manifest.
mistralai <huggingface.co/mistralai>Newsgroups: alt.freeweights.mistralSubject: 
mlx-community