CatalogWantedMirrorsLedgerOperating StatementDocsStudio statusCard system
alt.freeweights.deepseek / DeepSeek-R1-Distill-Llama-8B-GGUF
From: bartowski <huggingface.co/bartowski>Newsgroups: alt.freeweights.deepseekSubject:

DeepSeek-R1-Distill-Llama-8B-GGUF

Date: 2026-11-15Commit: 935d400Signed: 2DC28CC7B26A283B ✓

Text generation model. Gguf weights, 155 GB in all. Published by bartowski under MIT.

deepseek155GBDeepSeek-R1-Distill-Llama-8B-GGUFggufpermissive
FactsParamsnot statedSize155 GBFormatsggufTasktext generationLicenseMITFrom sourcepermissive
$ fw get bartowski/DeepSeek-R1-Distill-Llama-8B-GGUF
# manifest sha256 d805796c9ff8… · 155 GB · 27 files
$ fw verify ./DeepSeek-R1-Distill-Llama-8B-GGUF bartowski/DeepSeek-R1-Distill-Llama-8B-GGUF
From sourceHosted (signed)bartowski/DeepSeek-R1-Distill-Llama-8B-GGUFLast check: 2026-11-15 04:41 UTC
Run at homeGet help

Get help: custom fine-tune or private deploy. We review your setup before we quote.

Quantized of DeepSeek-R1-Distill-Llama-8B.

Works with

Run14Convert1Evaluate13Scan12Apps58

Convert with

ggml
OverviewModel cardFilesUse itLicenseProvenanceModel tree

Overview

Size
155 GB
Weights
2.95 GB (smallest of 24 GGUF variants)
Parameters
not stated
Formats
gguf
Task
text generation
License
MIT (permissive)
Files
27, each with a sha256
Publisher
bartowski
Added
2026-11-15
Section
alt.freeweights.deepseek
Lineage

Original release from the publisher.

Base model: deepseek-ai/DeepSeek-R1-Distill-Llama-8B

Where these bytes come from
Publisher at the pinned commit→You

Sources are tried in order. A file counts only when its size and sha256 match the manifest. Sources are checked when you ask for them, not on a timer.

Sources
BadgeHostLast checkResult
From sourcehuggingface.co2026-11-15 04:41 UTCpassed

Model card

From the publisher at commit 935d400. Shown as plain text with links; images open at the publisher.

Llamacpp imatrix Quantizations of DeepSeek-R1-Distill-Llama-8B#

Using llama.cpp release b4514 for quantization.

Original model: https://huggingface.co/deepseek-ai/DeepSeek-R1-Distill-Llama-8B

All quants made using imatrix option with dataset from here

Run them in LM Studio

Prompt format#

<｜begin▁of▁sentence｜>{system_prompt}<｜User｜>{prompt}<｜Assistant｜>

Download a file (not the whole branch) from below:#

FilenameQuant typeFile SizeSplitDescription
DeepSeek-R1-Distill-Llama-8B-f32.gguff3232.13GBfalseFull F32 weights.
DeepSeek-R1-Distill-Llama-8B-f16.gguff1616.07GBfalseFull F16 weights.
DeepSeek-R1-Distill-Llama-8B-Q8_0.ggufQ8_08.54GBfalseExtremely high quality, generally unneeded but max available quant.
DeepSeek-R1-Distill-Llama-8B-Q6_K_L.ggufQ6_K_L6.85GBfalseUses Q8_0 for embed and output weights. Very high quality, near perfect, recommended.
DeepSeek-R1-Distill-Llama-8B-Q6_K.ggufQ6_K6.60GBfalseVery high quality, near perfect, recommended.
DeepSeek-R1-Distill-Llama-8B-Q5_K_L.ggufQ5_K_L6.06GBfalseUses Q8_0 for embed and output weights. High quality, recommended.
DeepSeek-R1-Distill-Llama-8B-Q5_K_M.ggufQ5_K_M5.73GBfalseHigh quality, recommended.
DeepSeek-R1-Distill-Llama-8B-Q5_K_S.ggufQ5_K_S5.60GBfalseHigh quality, recommended.
DeepSeek-R1-Distill-Llama-8B-Q4_K_L.ggufQ4_K_L5.31GBfalseUses Q8_0 for embed and output weights. Good quality, recommended.
DeepSeek-R1-Distill-Llama-8B-Q4_1.ggufQ4_15.13GBfalseLegacy format, similar performance to Q4_K_S but with improved tokens/watt on Apple silicon.
DeepSeek-R1-Distill-Llama-8B-Q4_K_M.ggufQ4_K_M4.92GBfalseGood quality, default size for most use cases, recommended.
DeepSeek-R1-Distill-Llama-8B-Q3_K_XL.ggufQ3_K_XL4.78GBfalseUses Q8_0 for embed and output weights. Lower quality but usable, good for low RAM availability.
DeepSeek-R1-Distill-Llama-8B-Q4_K_S.ggufQ4_K_S4.69GBfalseSlightly lower quality with more space savings, recommended.
DeepSeek-R1-Distill-Llama-8B-Q4_0.ggufQ4_04.68GBfalseLegacy format, offers online repacking for ARM and AVX CPU inference.
DeepSeek-R1-Distill-Llama-8B-IQ4_NL.ggufIQ4_NL4.68GBfalseSimilar to IQ4_XS, but slightly larger. Offers online repacking for ARM CPU inference.
DeepSeek-R1-Distill-Llama-8B-IQ4_XS.ggufIQ4_XS4.45GBfalseDecent quality, smaller than Q4_K_S with similar performance, recommended.
DeepSeek-R1-Distill-Llama-8B-Q3_K_L.ggufQ3_K_L4.32GBfalseLower quality but usable, good for low RAM availability.
DeepSeek-R1-Distill-Llama-8B-Q3_K_M.ggufQ3_K_M4.02GBfalseLow quality.
DeepSeek-R1-Distill-Llama-8B-IQ3_M.ggufIQ3_M3.78GBfalseMedium-low quality, new method with decent performance comparable to Q3_K_M.
DeepSeek-R1-Distill-Llama-8B-Q2_K_L.ggufQ2_K_L3.69GBfalseUses Q8_0 for embed and output weights. Very low quality but surprisingly usable.
DeepSeek-R1-Distill-Llama-8B-Q3_K_S.ggufQ3_K_S3.66GBfalseLow quality, not recommended.
DeepSeek-R1-Distill-Llama-8B-IQ3_XS.ggufIQ3_XS3.52GBfalseLower quality, new method with decent performance, slightly better than Q3_K_S.
DeepSeek-R1-Distill-Llama-8B-Q2_K.ggufQ2_K3.18GBfalseVery low quality but surprisingly usable.
DeepSeek-R1-Distill-Llama-8B-IQ2_M.ggufIQ2_M2.95GBfalseRelatively low quality, uses SOTA techniques to be surprisingly usable.

Embed/output weights#

Some of these quants (Q3_K_XL, Q4_K_L etc) are the standard quantization method with the embeddings and output weights quantized to Q8_0 instead of what they would normally default to.

Downloading using huggingface-cli#

Click to view download instructions

First, make sure you have hugginface-cli installed:

pip install -U "huggingface_hub[cli]"

Then, you can target the specific file you want:

huggingface-cli download bartowski/DeepSeek-R1-Distill-Llama-8B-GGUF --include "DeepSeek-R1-Distill-Llama-8B-Q4_K_M.gguf" --local-dir ./

If the model is bigger than 50GB, it will have been split into multiple files. In order to download them all to a local folder, run:

huggingface-cli download bartowski/DeepSeek-R1-Distill-Llama-8B-GGUF --include "DeepSeek-R1-Distill-Llama-8B-Q8_0/*" --local-dir ./

You can either specify a new local-dir (DeepSeek-R1-Distill-Llama-8B-Q8_0) or download them all in place (./)

ARM/AVX information#

Previously, you would download Q4_0_4_4/4_8/8_8, and these would have their weights interleaved in memory in order to improve performance on ARM and AVX machines by loading up more data in one pass.

Now, however, there is something called "online repacking" for weights. details in this PR. If you use Q4_0 and your hardware would benefit from repacking weights, it will do it automatically on the fly.

As of llama.cpp build b4282 you will not be able to run the Q4_0_X_X files and will instead need to use Q4_0.

Additionally, if you want to get slightly better quality for , you can use IQ4_NL thanks to this PR which will also repack the weights for ARM, though only the 4_4 for now. The loading time may be slower but it will result in an overall speed incrase.

Click to view Q4_0_X_X information (deprecated

I'm keeping this section to show the potential theoretical uplift in performance from using the Q4_0 with online repacking.

Click to view benchmarks on an AVX2 system (EPYC7702)

modelsizeparamsbackendthreadstestt/s% (vs Q4_0)
qwen2 3B Q4_01.70 GiB3.09 BCPU64pp512204.03 ± 1.03100%
qwen2 3B Q4_01.70 GiB3.09 BCPU64pp1024282.92 ± 0.19100%
qwen2 3B Q4_01.70 GiB3.09 BCPU64pp2048259.49 ± 0.44100%
qwen2 3B Q4_01.70 GiB3.09 BCPU64tg12839.12 ± 0.27100%
qwen2 3B Q4_01.70 GiB3.09 BCPU64tg25639.31 ± 0.69100%
qwen2 3B Q4_01.70 GiB3.09 BCPU64tg51240.52 ± 0.03100%
qwen2 3B Q4_K_M1.79 GiB3.09 BCPU64pp512301.02 ± 1.74147%
qwen2 3B Q4_K_M1.79 GiB3.09 BCPU64pp1024287.23 ± 0.20101%
qwen2 3B Q4_K_M1.79 GiB3.09 BCPU64pp2048262.77 ± 1.81101%
qwen2 3B Q4_K_M1.79 GiB3.09 BCPU64tg12818.80 ± 0.9948%
qwen2 3B Q4_K_M1.79 GiB3.09 BCPU64tg25624.46 ± 3.0483%
qwen2 3B Q4_K_M1.79 GiB3.09 BCPU64tg51236.32 ± 3.5990%
qwen2 3B Q4_0_8_81.69 GiB3.09 BCPU64pp512271.71 ± 3.53133%
qwen2 3B Q4_0_8_81.69 GiB3.09 BCPU64pp1024279.86 ± 45.63100%
qwen2 3B Q4_0_8_81.69 GiB3.09 BCPU64pp2048320.77 ± 5.00124%
qwen2 3B Q4_0_8_81.69 GiB3.09 BCPU64tg12843.51 ± 0.05111%
qwen2 3B Q4_0_8_81.69 GiB3.09 BCPU64tg25643.35 ± 0.09110%
qwen2 3B Q4_0_8_81.69 GiB3.09 BCPU64tg51242.60 ± 0.31105%

Q4_0_8_8 offers a nice bump to prompt processing and a small bump to text generation

Which file should I choose?#

Click here for details

A great write up with charts showing various performances is provided by Artefact2 here

The first thing to figure out is how big a model you can run. To do this, you'll need to figure out how much RAM and/or VRAM you have.

If you want your model running as FAST as possible, you'll want to fit the whole thing on your GPU's VRAM. Aim for a quant with a file size 1-2GB smaller than your GPU's total VRAM.

If you want the absolute maximum quality, add both your system RAM and your GPU's VRAM together, then similarly grab a quant with a file size 1-2GB Smaller than that total.

Next, you'll need to decide if you want to use an 'I-quant' or a 'K-quant'.

If you don't want to think too much, grab one of the K-quants. These are in format 'QX_K_X', like Q5_K_M.

If you want to get more into the weeds, you can check out this extremely useful feature chart:

llama.cpp feature matrix

But basically, if you're aiming for below Q4, and you're running cuBLAS (Nvidia) or rocBLAS (AMD), you should look towards the I-quants. These are in format IQX_X, like IQ3_M. These are newer and offer better performance for their size.

These I-quants can also be used on CPU and Apple Metal, but will be slower than their K-quant equivalent, so speed vs performance is a tradeoff you'll have to decide.

The I-quants are not compatible with Vulcan, which is also AMD, so if you have an AMD card double check if you're using the rocBLAS build or the Vulcan build. At the time of writing this, LM Studio has a preview with ROCm support, and other inference engines have specific builds for ROCm.

Credits#

Thank you kalomaze and Dampf for assistance in creating the imatrix calibration dataset.

Thank you ZeroWw for the inspiration to experiment with embed/output.

Want to support my work? Visit my ko-fi page here: https://ko-fi.com/bartowski

Files

27 files · 155 GBWeights 2.95 GB (smallest of 24 GGUF variants). Sizes and sha256 from the signed manifest.
Show all 27 files with their sha256
FileSizesha256
.gitattributes3.42 KBcf006a9aef35d83c20c30c5150c400a00d6c03636cf1b68582daa530417cb634
DeepSeek-R1-Distill-Llama-8B-IQ2_M.gguf2.95 GB412416e0ab66254cb08ff5724e74e4ebc00ec311be0d4c51475e2ca95fe4e1ca
DeepSeek-R1-Distill-Llama-8B-IQ3_M.gguf3.78 GBba209188ab5db248d7033ab33b5f1044b48a1bc9886b85d5485304b1ca148f54
DeepSeek-R1-Distill-Llama-8B-IQ3_XS.gguf3.52 GB8b42e58abdc61ee61143e6ac41604ca4a78b9bc223045f56d3f8956c35145f81
DeepSeek-R1-Distill-Llama-8B-IQ4_NL.gguf4.68 GB8b25ad73b47fabb8689f9a6f57f707c792964aa8e1ce43fb7881efab22339b44
DeepSeek-R1-Distill-Llama-8B-IQ4_XS.gguf4.45 GBa076a5f7e48a054067d420c32740e1e63b425e1318a9a4eba89024906d67adcd
DeepSeek-R1-Distill-Llama-8B-Q2_K.gguf3.18 GB3a7e3487797b3b72bfb5506cddb510311c5f6070e1697499010b821edcc9bba5
DeepSeek-R1-Distill-Llama-8B-Q2_K_L.gguf3.69 GB0365f315c479670b71a1a4b03abe3bffa55afbec043e146488b1d9fba1c1cbea
DeepSeek-R1-Distill-Llama-8B-Q3_K_L.gguf4.32 GBd14b69b5c7be9b1485384f012f489fcacd2f3c52001ebe4d1ecf0fe0bcf51843
DeepSeek-R1-Distill-Llama-8B-Q3_K_M.gguf4.02 GB482c86c5ba38ce5cd19128cea057538aedad14fc4e5e1ffb9b112965b81a598b
DeepSeek-R1-Distill-Llama-8B-Q3_K_S.gguf3.66 GBedf43a1fbf6dd76d243d60a7b0335ec80b379885024db56ef71dca78e562f7ff
DeepSeek-R1-Distill-Llama-8B-Q3_K_XL.gguf4.78 GB40d880cc330d58bc8d6cf0cbafafbd1b1f783ecf2730479ffdc7de95f07e7906
DeepSeek-R1-Distill-Llama-8B-Q4_0.gguf4.68 GB0eb93e436ac8beec18aceb958c120d282cb2cf5451b23185e7be268fe9d375cc
DeepSeek-R1-Distill-Llama-8B-Q4_1.gguf5.13 GBd673d118541cabe72ce17deff6a0d34fcd97ec25ff9a8997575f2cb28395da43
DeepSeek-R1-Distill-Llama-8B-Q4_K_L.gguf5.31 GB75bfee2fb42bb1a53d2056380140b26edf320ef2eada7d6d412fa6ac530b90dc
DeepSeek-R1-Distill-Llama-8B-Q4_K_M.gguf4.92 GB87bcba20b4846d8dadf753d3ff48f9285d131fc95e3e0e7e934d4f20bc896f5d
DeepSeek-R1-Distill-Llama-8B-Q4_K_S.gguf4.69 GB89cfcce9b4d2855f0e6dfa8c6c14e664e7a45739e1a9eb1bc6265ebee41406a8
DeepSeek-R1-Distill-Llama-8B-Q5_K_L.gguf6.06 GBf224b08e2acaca8104bc4e1e5c259360a9c307bfe1235f3e2965c356606c6c42
DeepSeek-R1-Distill-Llama-8B-Q5_K_M.gguf5.73 GB645ceebaea72fddbc8941a9e1377e2b19f90513e53e7237d95d0b6069cb8add0
DeepSeek-R1-Distill-Llama-8B-Q5_K_S.gguf5.60 GBa2c223f7422d88fbcea40df7942961f741244062b46928ff2c2b59fba753569c
DeepSeek-R1-Distill-Llama-8B-Q6_K.gguf6.60 GB41bafde272416bb491448328dd4cf6d1bea03859eb2d3b90e846eb26e7b9f8d0
DeepSeek-R1-Distill-Llama-8B-Q6_K_L.gguf6.85 GB0e1fd1646250667de1aabb5bd952d8a5d5e32461dbabfd927c36729a4753d541
DeepSeek-R1-Distill-Llama-8B-Q8_0.gguf8.54 GBf22cd382e5829e82ad034887aa58fbadb929546191828a6a16a444a83089080e
DeepSeek-R1-Distill-Llama-8B-f16.gguf16.1 GB72b213c02ca41f2ca0933fb42f5872da74f291deeb43147f1bcd12ec06aa6185
DeepSeek-R1-Distill-Llama-8B-f32.gguf32.1 GB658978fc583fa6da4194bf9bd596bdc782aab527a6eae572c432a69751139982
DeepSeek-R1-Distill-Llama-8B.imatrix4.99 MB36a19b9ae01773cab755c5443c9b91a02f8b21eb6c2ab803673fc9f7842d9e48
README.md14.2 KB7cceac136dec35b7fc92453c7ddd03b3bfb2a5b804203a87c929a5b0d17d2c95
VersionsEach time the pinned upstream commit or file list changed in this repo. Newest first.
Upstream commitDateContents
935d400c2e pinned now2026-11-15 04:41 UTC27 files · 155 GB

Use it

$ fw get bartowski/DeepSeek-R1-Distill-Llama-8B-GGUF
# manifest sha256 d805796c9ff8… · 155 GB · 27 files
$ fw verify ./DeepSeek-R1-Distill-Llama-8B-GGUF bartowski/DeepSeek-R1-Distill-Llama-8B-GGUF
fw.pyChecks the signature and every sha256, tries each source in order, resumes.
$ python3 fw.py get https://freeweights.allternit.com/manifests/bartowski/DeepSeek-R1-Distill-Llama-8B-GGUF.json --dir ./DeepSeek-R1-Distill-Llama-8B-GGUF $ python3 fw.py verify ./DeepSeek-R1-Distill-Llama-8B-GGUF ./bartowski__DeepSeek-R1-Distill-Llama-8B-GGUF.json
hfThe Hugging Face CLI with one setting. Same repo id.
$ HF_ENDPOINT=https://freeweights.allternit.com hf download bartowski/DeepSeek-R1-Distill-Llama-8B-GGUF --revision 935d400c2ed1a29cc2a1045df25ec7998fb4dfe4 --local-dir ./DeepSeek-R1-Distill-Llama-8B-GGUF
llama.cppDownload one GGUF variant (DeepSeek-R1-Distill-Llama-8B-IQ2_M, 2.95 GB), then serve it.
$ HF_ENDPOINT=https://freeweights.allternit.com hf download bartowski/DeepSeek-R1-Distill-Llama-8B-GGUF --include "DeepSeek-R1-Distill-Llama-8B-IQ2_M*" --local-dir ./DeepSeek-R1-Distill-Llama-8B-GGUF $ llama-server -m ./DeepSeek-R1-Distill-Llama-8B-GGUF/DeepSeek-R1-Distill-Llama-8B-IQ2_M.gguf
OllamaMake an Ollama model from the downloaded file.
$ printf 'FROM ./DeepSeek-R1-Distill-Llama-8B-GGUF/DeepSeek-R1-Distill-Llama-8B-IQ2_M.gguf\n' > Modelfile $ ollama create deepseek-r1-distill-llama-8b-gguf -f Modelfile
Run at home

License

Name
MIT License
Id
MIT
Class
Permissive
Redistribution
yes
Listing
Hosted (signed)
Decided
Inherited from the base model deepseek-ai/DeepSeek-R1-Distill-Llama-8B: the strictest license along the lineage applies. 2026-11-15
Text in
README.md
Text sha256
7cceac136dec35b7fc92453c7ddd03b3bfb2a5b804203a87c929a5b0d17d2c95
Terms gate
No extra terms beyond the license.

Read the license text

Licenses are checked against the license table. A custom license passes only after a person reads it.

Provenance

From: bartowski <huggingface.co/bartowski>Newsgroups: alt.freeweights.deepseekSubject: DeepSeek-R1-Distill-Llama-8B-GGUFDate: 2026-11-15Commit: 935d400Signed: 2DC28CC7B26A283B ✓
Upstream
bartowski/DeepSeek-R1-Distill-Llama-8B-GGUF
Pinned commit
935d400c2ed1a29cc2a1045df25ec7998fb4dfe4
Manifest
DeepSeek-R1-Distill-Llama-8B-GGUF.json · .minisig
Manifest sha256
d805796c9ff84249287963590ea71f975bfebf032ea23de1beabbc4caa7f3fa2
Signature
minisign, Ed25519, key 2DC28CC7B26A283B
Created
2026-11-15 04:41 UTC
Generator
freeweights-manifest/0.1.0
Takedowns
None.

How to verify

$ curl -O https://freeweights.allternit.com/manifests/bartowski/DeepSeek-R1-Distill-Llama-8B-GGUF.json $ curl -O https://freeweights.allternit.com/manifests/bartowski/DeepSeek-R1-Distill-Llama-8B-GGUF.json.minisig $ minisign -Vm DeepSeek-R1-Distill-Llama-8B-GGUF.json -P RWQ7KGqyx4zCLXtUFOqvfv8L3g+8fpKJ1FsuxPyOYMF+4KufbUxKq1Oq $ shasum -a 256 <file> # compare with the Files tab

A manifest whose signature fails must be rejected. A file whose size or sha256 does not match must be discarded. Verify a download

Model tree

Parent

deepseek-ai/DeepSeek-R1-Distill-Llama-8B · this model is a quantization of it

No model in the catalog names this one as its parent.

Get it
Download
$ HF_ENDPOINT=https://freeweights.allternit.com hf download bartowski/DeepSeek-R1-Distill-Llama-8B-GGUF
Open in Freeweights Studio

Pull, verify and run it in the Allternit app on Desktop, web or phone.

Signed manifest
minisign, key 2DC28CC7B26A283B. DeepSeek-R1-Distill-Llama-8B-GGUF.json. Provenance. Takedown history: none.
Source
huggingface.co/bartowski/DeepSeek-R1-Distill-Llama-8B-GGUF

Related models

All of alt.freeweights.deepseek →
From: timm <huggingface.co/timm>Subject: deepseek_vit_412m.deepseek_v4_1_flash
From: bartowski <huggingface.co/bartowski>Subject: deepseek-ai_DeepSeek-R1-0528-GGUF
From: bartowski <huggingface.co/bartowski>Subject: deepseek-ai_DeepSeek-R1-0528-Qwen3-8B-GGUF
From: bartowski <huggingface.co/bartowski>Subject: deepseek-ai_DeepSeek-V3-0324-GGUF
From: bartowski <huggingface.co/bartowski>Subject: deepseek-ai_DeepSeek-V3.1-GGUF
From: mlx-community <huggingface.co/mlx-community>Subject: deepseek-ai-DeepSeek-V4-Flash-8bit