CatalogWantedMirrorsLedgerOperating StatementDocsStudio statusCard system
alt.freeweights.deepseek / DeepSeek-R1-Distill-Llama-70B-GGUF
From: bartowski <huggingface.co/bartowski>Newsgroups: alt.freeweights.deepseekSubject:

DeepSeek-R1-Distill-Llama-70B-GGUF

Date: 2026-11-14Commit: 1842c5fSigned: 2DC28CC7B26A283B ✓

Text generation model. Gguf weights, 834 GB in all. Published by bartowski under MIT.

deepseek834GBDeepSeek-R1-Distill-Llama-70B-GGUFggufpermissive
FactsParamsnot statedSize834 GBFormatsggufTasktext generationLicenseMITFrom sourcepermissive
$ fw get bartowski/DeepSeek-R1-Distill-Llama-70B-GGUF
# manifest sha256 e7761fdf22d8… · 834 GB · 29 files
$ fw verify ./DeepSeek-R1-Distill-Llama-70B-GGUF bartowski/DeepSeek-R1-Distill-Llama-70B-GGUF
From sourceHosted (signed)bartowski/DeepSeek-R1-Distill-Llama-70B-GGUFLast check: 2026-11-14 21:27 UTC
Run at homeGet help

Get help: custom fine-tune or private deploy. We review your setup before we quote.

Quantized of DeepSeek-R1-Distill-Llama-70B.

Works with

Run14Convert1Evaluate13Scan12Apps58

Convert with

ggml
OverviewModel cardFilesUse itLicenseProvenanceModel tree

Overview

Size
834 GB
Weights
16.8 GB (smallest of 23 GGUF variants)
Parameters
not stated
Formats
gguf
Task
text generation
License
MIT (permissive)
Files
29, each with a sha256
Publisher
bartowski
Added
2026-11-14
Section
alt.freeweights.deepseek
Lineage

Original release from the publisher.

Base model: deepseek-ai/DeepSeek-R1-Distill-Llama-70B

Where these bytes come from
Publisher at the pinned commit→You

Sources are tried in order. A file counts only when its size and sha256 match the manifest. Sources are checked when you ask for them, not on a timer.

Sources
BadgeHostLast checkResult
From sourcehuggingface.co2026-11-14 21:27 UTCpassed

Model card

From the publisher at commit 1842c5f. Shown as plain text with links; images open at the publisher.

Llamacpp imatrix Quantizations of DeepSeek-R1-Distill-Llama-70B#

Using llama.cpp release b4514 for quantization.

Original model: https://huggingface.co/deepseek-ai/DeepSeek-R1-Distill-Llama-70B

All quants made using imatrix option with dataset from here

Run them in LM Studio

Prompt format#

<｜begin▁of▁sentence｜>{system_prompt}<｜User｜>{prompt}<｜Assistant｜>

Download a file (not the whole branch) from below:#

FilenameQuant typeFile SizeSplitDescription
DeepSeek-R1-Distill-Llama-70B-Q8_0.ggufQ8_074.98GBtrueExtremely high quality, generally unneeded but max available quant.
DeepSeek-R1-Distill-Llama-70B-Q6_K.ggufQ6_K57.89GBtrueVery high quality, near perfect, recommended.
DeepSeek-R1-Distill-Llama-70B-Q5_K_M.ggufQ5_K_M49.95GBtrueHigh quality, recommended.
DeepSeek-R1-Distill-Llama-70B-Q5_K_S.ggufQ5_K_S48.66GBfalseHigh quality, recommended.
DeepSeek-R1-Distill-Llama-70B-Q4_1.ggufQ4_144.31GBfalseLegacy format, similar performance to Q4_K_S but with improved tokens/watt on Apple silicon.
DeepSeek-R1-Distill-Llama-70B-Q4_K_M.ggufQ4_K_M42.52GBfalseGood quality, default size for most use cases, recommended.
DeepSeek-R1-Distill-Llama-70B-Q4_K_S.ggufQ4_K_S40.35GBfalseSlightly lower quality with more space savings, recommended.
DeepSeek-R1-Distill-Llama-70B-Q4_0.ggufQ4_040.12GBfalseLegacy format, offers online repacking for ARM and AVX CPU inference.
DeepSeek-R1-Distill-Llama-70B-IQ4_NL.ggufIQ4_NL40.05GBfalseSimilar to IQ4_XS, but slightly larger. Offers online repacking for ARM CPU inference.
DeepSeek-R1-Distill-Llama-70B-Q3_K_XL.ggufQ3_K_XL38.06GBfalseUses Q8_0 for embed and output weights. Lower quality but usable, good for low RAM availability.
DeepSeek-R1-Distill-Llama-70B-IQ4_XS.ggufIQ4_XS37.90GBfalseDecent quality, smaller than Q4_K_S with similar performance, recommended.
DeepSeek-R1-Distill-Llama-70B-Q3_K_L.ggufQ3_K_L37.14GBfalseLower quality but usable, good for low RAM availability.
DeepSeek-R1-Distill-Llama-70B-Q3_K_M.ggufQ3_K_M34.27GBfalseLow quality.
DeepSeek-R1-Distill-Llama-70B-IQ3_M.ggufIQ3_M31.94GBfalseMedium-low quality, new method with decent performance comparable to Q3_K_M.
DeepSeek-R1-Distill-Llama-70B-Q3_K_S.ggufQ3_K_S30.91GBfalseLow quality, not recommended.
DeepSeek-R1-Distill-Llama-70B-IQ3_XXS.ggufIQ3_XXS27.47GBfalseLower quality, new method with decent performance, comparable to Q3 quants.
DeepSeek-R1-Distill-Llama-70B-Q2_K_L.ggufQ2_K_L27.40GBfalseUses Q8_0 for embed and output weights. Very low quality but surprisingly usable.
DeepSeek-R1-Distill-Llama-70B-Q2_K.ggufQ2_K26.38GBfalseVery low quality but surprisingly usable.
DeepSeek-R1-Distill-Llama-70B-IQ2_M.ggufIQ2_M24.12GBfalseRelatively low quality, uses SOTA techniques to be surprisingly usable.
DeepSeek-R1-Distill-Llama-70B-IQ2_S.ggufIQ2_S22.24GBfalseLow quality, uses SOTA techniques to be usable.
DeepSeek-R1-Distill-Llama-70B-IQ2_XS.ggufIQ2_XS21.14GBfalseLow quality, uses SOTA techniques to be usable.
DeepSeek-R1-Distill-Llama-70B-IQ2_XXS.ggufIQ2_XXS19.10GBfalseVery low quality, uses SOTA techniques to be usable.
DeepSeek-R1-Distill-Llama-70B-IQ1_M.ggufIQ1_M16.75GBfalseExtremely low quality, not recommended.

Embed/output weights#

Some of these quants (Q3_K_XL, Q4_K_L etc) are the standard quantization method with the embeddings and output weights quantized to Q8_0 instead of what they would normally default to.

Downloading using huggingface-cli#

Click to view download instructions

First, make sure you have hugginface-cli installed:

pip install -U "huggingface_hub[cli]"

Then, you can target the specific file you want:

huggingface-cli download bartowski/DeepSeek-R1-Distill-Llama-70B-GGUF --include "DeepSeek-R1-Distill-Llama-70B-Q4_K_M.gguf" --local-dir ./

If the model is bigger than 50GB, it will have been split into multiple files. In order to download them all to a local folder, run:

huggingface-cli download bartowski/DeepSeek-R1-Distill-Llama-70B-GGUF --include "DeepSeek-R1-Distill-Llama-70B-Q8_0/*" --local-dir ./

You can either specify a new local-dir (DeepSeek-R1-Distill-Llama-70B-Q8_0) or download them all in place (./)

ARM/AVX information#

Previously, you would download Q4_0_4_4/4_8/8_8, and these would have their weights interleaved in memory in order to improve performance on ARM and AVX machines by loading up more data in one pass.

Now, however, there is something called "online repacking" for weights. details in this PR. If you use Q4_0 and your hardware would benefit from repacking weights, it will do it automatically on the fly.

As of llama.cpp build b4282 you will not be able to run the Q4_0_X_X files and will instead need to use Q4_0.

Additionally, if you want to get slightly better quality for , you can use IQ4_NL thanks to this PR which will also repack the weights for ARM, though only the 4_4 for now. The loading time may be slower but it will result in an overall speed incrase.

Click to view Q4_0_X_X information (deprecated

I'm keeping this section to show the potential theoretical uplift in performance from using the Q4_0 with online repacking.

Click to view benchmarks on an AVX2 system (EPYC7702)

modelsizeparamsbackendthreadstestt/s% (vs Q4_0)
qwen2 3B Q4_01.70 GiB3.09 BCPU64pp512204.03 ± 1.03100%
qwen2 3B Q4_01.70 GiB3.09 BCPU64pp1024282.92 ± 0.19100%
qwen2 3B Q4_01.70 GiB3.09 BCPU64pp2048259.49 ± 0.44100%
qwen2 3B Q4_01.70 GiB3.09 BCPU64tg12839.12 ± 0.27100%
qwen2 3B Q4_01.70 GiB3.09 BCPU64tg25639.31 ± 0.69100%
qwen2 3B Q4_01.70 GiB3.09 BCPU64tg51240.52 ± 0.03100%
qwen2 3B Q4_K_M1.79 GiB3.09 BCPU64pp512301.02 ± 1.74147%
qwen2 3B Q4_K_M1.79 GiB3.09 BCPU64pp1024287.23 ± 0.20101%
qwen2 3B Q4_K_M1.79 GiB3.09 BCPU64pp2048262.77 ± 1.81101%
qwen2 3B Q4_K_M1.79 GiB3.09 BCPU64tg12818.80 ± 0.9948%
qwen2 3B Q4_K_M1.79 GiB3.09 BCPU64tg25624.46 ± 3.0483%
qwen2 3B Q4_K_M1.79 GiB3.09 BCPU64tg51236.32 ± 3.5990%
qwen2 3B Q4_0_8_81.69 GiB3.09 BCPU64pp512271.71 ± 3.53133%
qwen2 3B Q4_0_8_81.69 GiB3.09 BCPU64pp1024279.86 ± 45.63100%
qwen2 3B Q4_0_8_81.69 GiB3.09 BCPU64pp2048320.77 ± 5.00124%
qwen2 3B Q4_0_8_81.69 GiB3.09 BCPU64tg12843.51 ± 0.05111%
qwen2 3B Q4_0_8_81.69 GiB3.09 BCPU64tg25643.35 ± 0.09110%
qwen2 3B Q4_0_8_81.69 GiB3.09 BCPU64tg51242.60 ± 0.31105%

Q4_0_8_8 offers a nice bump to prompt processing and a small bump to text generation

Which file should I choose?#

Click here for details

A great write up with charts showing various performances is provided by Artefact2 here

The first thing to figure out is how big a model you can run. To do this, you'll need to figure out how much RAM and/or VRAM you have.

If you want your model running as FAST as possible, you'll want to fit the whole thing on your GPU's VRAM. Aim for a quant with a file size 1-2GB smaller than your GPU's total VRAM.

If you want the absolute maximum quality, add both your system RAM and your GPU's VRAM together, then similarly grab a quant with a file size 1-2GB Smaller than that total.

Next, you'll need to decide if you want to use an 'I-quant' or a 'K-quant'.

If you don't want to think too much, grab one of the K-quants. These are in format 'QX_K_X', like Q5_K_M.

If you want to get more into the weeds, you can check out this extremely useful feature chart:

llama.cpp feature matrix

But basically, if you're aiming for below Q4, and you're running cuBLAS (Nvidia) or rocBLAS (AMD), you should look towards the I-quants. These are in format IQX_X, like IQ3_M. These are newer and offer better performance for their size.

These I-quants can also be used on CPU and Apple Metal, but will be slower than their K-quant equivalent, so speed vs performance is a tradeoff you'll have to decide.

The I-quants are not compatible with Vulcan, which is also AMD, so if you have an AMD card double check if you're using the rocBLAS build or the Vulcan build. At the time of writing this, LM Studio has a preview with ROCm support, and other inference engines have specific builds for ROCm.

Credits#

Thank you kalomaze and Dampf for assistance in creating the imatrix calibration dataset.

Thank you ZeroWw for the inspiration to experiment with embed/output.

Want to support my work? Visit my ko-fi page here: https://ko-fi.com/bartowski

Files

29 files · 834 GBWeights 16.8 GB (smallest of 23 GGUF variants). Sizes and sha256 from the signed manifest.
Show all 29 files with their sha256
FileSizesha256
.gitattributes3.91 KBfd0648c766894e8da5018368fba0001ea79f5eefbe7f2470b4ea5aff5058ba96
DeepSeek-R1-Distill-Llama-70B-IQ1_M.gguf16.8 GB2ceb17221a51292dac93fa2518bb24ee4a9f8723df9cca29201649b30492f6ba
DeepSeek-R1-Distill-Llama-70B-IQ2_M.gguf24.1 GB614a5cb355ba58883ed6a2a8b4950d9d3c9571e90ff67450c4e14e68bf6cb222
DeepSeek-R1-Distill-Llama-70B-IQ2_S.gguf22.2 GBbec2f0d2516837ec4ed671f57b11e2dfebd4d89fd46f78c4caf064f5fd4969c7
DeepSeek-R1-Distill-Llama-70B-IQ2_XS.gguf21.1 GB5191033aacdc857c0f39e5d1e46bcc55f1b9f88074b00c86b7bda229a711d403
DeepSeek-R1-Distill-Llama-70B-IQ2_XXS.gguf19.1 GB44a8d751ec875182ba40e77a3f35d363cf60f1eb6c8f96d9fa4603f5222b9e47
DeepSeek-R1-Distill-Llama-70B-IQ3_M.gguf31.9 GB31cd7e6241ade7f4bddeb1ab13b23ff56932839b76d09c8cec925dc376d0d575
DeepSeek-R1-Distill-Llama-70B-IQ3_XXS.gguf27.5 GB128740ac2c0d093b3a50e1812c51e2ef28094da356a6f6a5b0a335b9f95ef620
DeepSeek-R1-Distill-Llama-70B-IQ4_NL.gguf40.1 GB921c814392aa411f6de8f7e8a9214887b0921d3809aa9f0c571ae7dec8ffc9fc
DeepSeek-R1-Distill-Llama-70B-IQ4_XS.gguf37.9 GB6e72d7a46a857b7d2e05dda32ee6366eff548bc73f07f6b112e88ff10d3467e5
DeepSeek-R1-Distill-Llama-70B-Q2_K.gguf26.4 GBa320278a17145cbfa48120d12bfacbb512165104f8f0b8cf180b03ea9549ba53
DeepSeek-R1-Distill-Llama-70B-Q2_K_L.gguf27.4 GB5cb5412a87b6331a2945c6b49c504fa549a760877d7e0d48a40368240b858652
DeepSeek-R1-Distill-Llama-70B-Q3_K_L.gguf37.1 GB3a0a66c801fe6fc749dd015f6903bdafbb3b9b42ab878cd56bd15c9f6cea7ff6
DeepSeek-R1-Distill-Llama-70B-Q3_K_M.gguf34.3 GB3778a253baf5188c056218430eaab3d8ad913ca72a33494c432c536f67349c95
DeepSeek-R1-Distill-Llama-70B-Q3_K_S.gguf30.9 GB803d1711948c8448002e166e09c44210831123d4bf0790130484af3cc4e8d4cb
DeepSeek-R1-Distill-Llama-70B-Q3_K_XL.gguf38.1 GBf2066821111a5a154f34332cf9e54e3da327293747b7f9330bdcdddf7022763e
DeepSeek-R1-Distill-Llama-70B-Q4_0.gguf40.1 GBe83a4211707edd912796495927b24ebf77711c6e4160a844c583d76285f6c712
DeepSeek-R1-Distill-Llama-70B-Q4_1.gguf44.3 GB5f9a3e34a70d2260d91951750313a043898c097965543ee345a276da1ccb9531
DeepSeek-R1-Distill-Llama-70B-Q4_K_M.gguf42.5 GB181a82a1d6d2fa24fe4db83a68eee030384986bdbdd4773ba76424e3a6eb9fd8
DeepSeek-R1-Distill-Llama-70B-Q4_K_S.gguf40.3 GB8b4855bf633aca0f9947d7e4b218bc847090ddb41278a30b0951d0f1c1d4c8e6
DeepSeek-R1-Distill-Llama-70B-Q5_K_M/DeepSeek-R1-Distill-Llama-70B-Q5_K_M-00001-of-00002.gguf40.0 GBfa5da9538c5e2fe599f6d330e2fc09152b22e6f89c0292dbcf4f58838bff54c5
DeepSeek-R1-Distill-Llama-70B-Q5_K_M/DeepSeek-R1-Distill-Llama-70B-Q5_K_M-00002-of-00002.gguf9.96 GBaf0c9e714ad2ac2297a4087ad97ef8475bb6a8821f364f63f424221cec68c8fe
DeepSeek-R1-Distill-Llama-70B-Q5_K_S.gguf48.7 GB3b170b8a701c55850036a5205d8477b68ac59f34b6ecc28e3c7a9b9b486cbadf
DeepSeek-R1-Distill-Llama-70B-Q6_K/DeepSeek-R1-Distill-Llama-70B-Q6_K-00001-of-00002.gguf40.0 GBa0bccdec0e04ea38de33acfa614dcb3d4394df53954e2bd81fefec0b2ef74351
DeepSeek-R1-Distill-Llama-70B-Q6_K/DeepSeek-R1-Distill-Llama-70B-Q6_K-00002-of-00002.gguf17.9 GB159bf34c24a7a1ca2a3b0563d534e7e026f8b389ebe88051f92a232c20ed83da
DeepSeek-R1-Distill-Llama-70B-Q8_0/DeepSeek-R1-Distill-Llama-70B-Q8_0-00001-of-00002.gguf39.9 GB74e77eeeae9d2348979299ff1e495477c01c8f9d93989fdd7319fbcf2115e441
DeepSeek-R1-Distill-Llama-70B-Q8_0/DeepSeek-R1-Distill-Llama-70B-Q8_0-00002-of-00002.gguf35.0 GB74b567c55fb8d9b9eb958b871f75f55e20d818e51974fb2540cfc33025c0ed7a
DeepSeek-R1-Distill-Llama-70B.imatrix24.9 MBe2e8aacf84d3aec9e108d06452f71d190cadff50403be793dd36d72f243b4589
README.md14.0 KB7cb92771750fbe39d602712a1221b221e8b7bf48836f9baa505a80b68394a5de
VersionsEach time the pinned upstream commit or file list changed in this repo. Newest first.
Upstream commitDateContents
1842c5f728 pinned now manifest updated, same commit2026-11-14 21:27 UTC29 files · 834 GB
1842c5f7282026-11-14 21:27 UTC29 files · 834 GB

Use it

$ fw get bartowski/DeepSeek-R1-Distill-Llama-70B-GGUF
# manifest sha256 e7761fdf22d8… · 834 GB · 29 files
$ fw verify ./DeepSeek-R1-Distill-Llama-70B-GGUF bartowski/DeepSeek-R1-Distill-Llama-70B-GGUF
fw.pyChecks the signature and every sha256, tries each source in order, resumes.
$ python3 fw.py get https://freeweights.allternit.com/manifests/bartowski/DeepSeek-R1-Distill-Llama-70B-GGUF.json --dir ./DeepSeek-R1-Distill-Llama-70B-GGUF $ python3 fw.py verify ./DeepSeek-R1-Distill-Llama-70B-GGUF ./bartowski__DeepSeek-R1-Distill-Llama-70B-GGUF.json
hfThe Hugging Face CLI with one setting. Same repo id.
$ HF_ENDPOINT=https://freeweights.allternit.com hf download bartowski/DeepSeek-R1-Distill-Llama-70B-GGUF --revision 1842c5f7280f933ead58adf8afd078672c9f6cd0 --local-dir ./DeepSeek-R1-Distill-Llama-70B-GGUF
llama.cppDownload one GGUF variant (DeepSeek-R1-Distill-Llama-70B-IQ1_M, 16.8 GB), then serve it.
$ HF_ENDPOINT=https://freeweights.allternit.com hf download bartowski/DeepSeek-R1-Distill-Llama-70B-GGUF --include "DeepSeek-R1-Distill-Llama-70B-IQ1_M*" --local-dir ./DeepSeek-R1-Distill-Llama-70B-GGUF $ llama-server -m ./DeepSeek-R1-Distill-Llama-70B-GGUF/DeepSeek-R1-Distill-Llama-70B-IQ1_M.gguf
OllamaMake an Ollama model from the downloaded file.
$ printf 'FROM ./DeepSeek-R1-Distill-Llama-70B-GGUF/DeepSeek-R1-Distill-Llama-70B-IQ1_M.gguf\n' > Modelfile $ ollama create deepseek-r1-distill-llama-70b-gguf -f Modelfile
Run at home

License

Name
MIT License
Id
MIT
Class
Permissive
Redistribution
yes
Listing
Hosted (signed)
Decided
Inherited from the base model deepseek-ai/DeepSeek-R1-Distill-Llama-70B: the strictest license along the lineage applies. 2026-11-14
Text in
README.md
Text sha256
7cb92771750fbe39d602712a1221b221e8b7bf48836f9baa505a80b68394a5de
Terms gate
No extra terms beyond the license.

Read the license text

Licenses are checked against the license table. A custom license passes only after a person reads it.

Provenance

From: bartowski <huggingface.co/bartowski>Newsgroups: alt.freeweights.deepseekSubject: DeepSeek-R1-Distill-Llama-70B-GGUFDate: 2026-11-14Commit: 1842c5fSigned: 2DC28CC7B26A283B ✓
Upstream
bartowski/DeepSeek-R1-Distill-Llama-70B-GGUF
Pinned commit
1842c5f7280f933ead58adf8afd078672c9f6cd0
Manifest
DeepSeek-R1-Distill-Llama-70B-GGUF.json · .minisig
Manifest sha256
e7761fdf22d8d2cbb30de1756619891d5ec05ceebc97753af06ae5f1305c3def
Signature
minisign, Ed25519, key 2DC28CC7B26A283B
Created
2026-11-14 21:27 UTC
Generator
freeweights-manifest/0.1.0
Takedowns
None.

How to verify

$ curl -O https://freeweights.allternit.com/manifests/bartowski/DeepSeek-R1-Distill-Llama-70B-GGUF.json $ curl -O https://freeweights.allternit.com/manifests/bartowski/DeepSeek-R1-Distill-Llama-70B-GGUF.json.minisig $ minisign -Vm DeepSeek-R1-Distill-Llama-70B-GGUF.json -P RWQ7KGqyx4zCLXtUFOqvfv8L3g+8fpKJ1FsuxPyOYMF+4KufbUxKq1Oq $ shasum -a 256 <file> # compare with the Files tab

A manifest whose signature fails must be rejected. A file whose size or sha256 does not match must be discarded. Verify a download

Model tree

Parent

deepseek-ai/DeepSeek-R1-Distill-Llama-70B · this model is a quantization of it

No model in the catalog names this one as its parent.

Get it
Download
$ HF_ENDPOINT=https://freeweights.allternit.com hf download bartowski/DeepSeek-R1-Distill-Llama-70B-GGUF
Open in Freeweights Studio

Pull, verify and run it in the Allternit app on Desktop, web or phone.

Signed manifest
minisign, key 2DC28CC7B26A283B. DeepSeek-R1-Distill-Llama-70B-GGUF.json. Provenance. Takedown history: none.
Source
huggingface.co/bartowski/DeepSeek-R1-Distill-Llama-70B-GGUF

Related models

All of alt.freeweights.deepseek →
From: timm <huggingface.co/timm>Subject: deepseek_vit_412m.deepseek_v4_1_flash
From: bartowski <huggingface.co/bartowski>Subject: deepseek-ai_DeepSeek-R1-0528-GGUF
From: bartowski <huggingface.co/bartowski>Subject: deepseek-ai_DeepSeek-R1-0528-Qwen3-8B-GGUF
From: bartowski <huggingface.co/bartowski>Subject: deepseek-ai_DeepSeek-V3-0324-GGUF
From: bartowski <huggingface.co/bartowski>Subject: deepseek-ai_DeepSeek-V3.1-GGUF
From: mlx-community <huggingface.co/mlx-community>Subject: deepseek-ai-DeepSeek-V4-Flash-8bit