From:

huggingface <github.com/huggingface>Newsgroups: alt.freeweights.toolsSubject:

datatrove

Release: v0.10.1 (2026-09-30)Commit: 1977fbbLicense: Apache-2.0

Freeing data processing from scripting madness by providing a set of platform-agnostic customizable pipeline processing blocks.

data and documents

v0.10.1releasedatatrovePythonApache-2.0

FactsLicenseApache-2.0LanguagePythonKinddata and documentsRelease2026-09-30PlatformsLinux, macOS, WindowsLinks to the projectpermissive

$ git clone git@github.com:huggingface/datatrove.git && cd datatrove

Overview Install Use with Freeweights License Provenance

Overview

Filters, deduplicates and processes very large text datasets in pipelines.

Kind: data and documents
License: Apache-2.0
Language: Python
Platforms: Linux, macOS, Windows
Latest release: v0.10.1 (2026-09-30)
Repository: huggingface/datatrove

Freeweights links to the project and does not host tool binaries.

Install

$ git clone git@github.com:huggingface/datatrove.git && cd datatrove

$ git clone git@github.com:huggingface/datatrove.git && cd datatrove

Checked against the project's README on 2026-11-15. The project's own instructions win when they differ: install docs

Freeweights links to the project and does not host tool binaries.

Use with Freeweights

Use a catalog model

Download the weights from Freeweights first (HF_ENDPOINT or fw.py), then point the tool at the folder.

Catalog models it can run (10)

code_generation_lite tulu-3-sft-mixture fineweb-edu gsm8k hh-rlhf MMLU-Pro oasst2 openai_humaneval ultrafeedback_binarized WildChat-4.8M

License

License: Apache-2.0
Class: Permissive

Read the license

Freeweights links to the project and does not host tool binaries.

Provenance

From:

huggingface <github.com/huggingface>Newsgroups: alt.freeweights.toolsSubject: datatroveRelease: v0.10.1 (2026-09-30)Commit: 1977fbbLicense: Apache-2.0

Repository: huggingface/datatrove
Pinned release: v0.10.1, 2026-09-30
Release commit: 1977fbb0f3c163d43cace334b073dda17fa3ef26
Default branch: main 5484e67ab81daf66c689106054d5133910d731c2
Fetched: 2026-11-15
Takedowns: None.

Freeweights links to the project and does not host tool binaries.

Get it

Install Project repository

Works with

10 catalog models. See the list