From: stanford-crfm <github.com/stanford-crfm>Newsgroups: alt.freeweights.toolsSubject:

HELM

Release: v0.5.16 (2026-04-30)Commit: ef439fcLicense: Apache-2.0

Holistic Evaluation of Language Models (HELM) is an open source Python framework created by the Center for Research on Foundation Models (CRFM) at Stanford for holistic, reproducible and transparent evaluation of foundation models, including large language models (LLMs) and multimodal models.

evaluationv0.5.16releaseHELMPythonApache-2.0

FactsLicenseApache-2.0LanguagePythonKindevaluationRelease2026-04-30PlatformsLinux, macOS, WindowsLinks to the projectpermissive

$ pip install crfm-helm

Overview Install Use with Freeweights License Provenance

Overview

Stanford's holistic evaluation of language models across scenarios and metrics.

Kind: evaluation
License: Apache-2.0
Language: Python
Platforms: Linux, macOS, Windows
Latest release: v0.5.16 (2026-04-30)
Repository: stanford-crfm/helm

Freeweights links to the project and does not host tool binaries.

Install

$ pip install crfm-helm

$ pip install crfm-helm

Checked against the project's README on 2026-11-14. The project's own instructions win when they differ: install docs

Freeweights links to the project and does not host tool binaries.

Use with Freeweights

Evaluate a catalog model

Download the weights from Freeweights first (HF_ENDPOINT or fw.py), then point the tool at the folder.

Catalog models it can run (4926)

135M-lc-100k-rope-12B a2-rl-e2egit_large-50-32B Accio-Lab_occamy-1.0-GGUF AesCode-32B AesCode-32B-GGUF AesCode-8B AesCode-8B-GGUF AFM-4.5B AFM-4.5B-Base AFM-4.5B-ONNX AgentCPM-Report agentica-org_DeepCoder-14B-Preview-GGUF agentica-org_DeepScaleR-1.5B-Preview-GGUF agentica-org_DeepSWE-Preview-GGUF Agents-A1-4B AI21-Jamba-Reasoning-3B AI21-Jamba2-Mini ai21labs_AI21-Jamba-Reasoning-3B-GGUF ai21labs_AI21-Jamba2-Mini-GGUF ai9stars_G9v3-3B-GGUF alamios_Mistral-Small-3.1-DRAFT-0.5B-GGUF AlchemistCoder-L-7B AlchemistCoder-L-7B-GGUF Alexandria-Gemma-4-12B-it-Ornith9-Summaries-LoRA-r128 Alexandria-Gemma-4-12B-it-Ornith9-Summaries-LoRA-r64 Alexandria-Gemma-4-12B-it-Qwen27-Knowledge-Units-LoRA-r128 Alexandria-Gemma-4-12B-it-Qwen27-Summaries-LoRA-r128 Alexandria-Gemma-4-12B-it-Qwen27-Summaries-LoRA-r64 Alexandria-Gemma-4-E4B-it-Qwen27-Knowledge-Units-LoRA-r128 Alibaba-NLP_Tongyi-DeepResearch-30B-A3B-GGUF ALLaM-7B-Instruct-preview allenai_Olmo-3-7B-Instruct-GGUF allenai_Olmo-3-7B-Think-GGUF allenai_Olmo-3.1-32B-Instruct-GGUF allenai_Olmo-3.1-32B-Think-GGUF allenai_olmOCR-2-7B-1025-GGUF allenai_olmOCR-7B-0225-preview-GGUF AM-Thinking-v1 AM-Thinking-v1-GGUF AMD-OLMo-1B AMD-OLMo-1B AMD-OLMo-1B-SFT AMD-OLMo-1B-SFT AMD-OLMo-1B-SFT-DPO AMD-OLMo-1B-SFT-DPO AMD-OLMo-1B-SFT-DPO-GGUF AMD-OLMo-1B-SFT-GGUF Apertus-70B-2509 Apertus-70B-Instruct-2509 Apertus-70B-Instruct-2509-GGUF Apertus-8B-2509 Apertus-8B-Instruct-2509 Apertus-8B-Instruct-2509-GGUF Apertus-8B-Instruct-2509-ONNX Apertus-8B-Instruct-2509-unsloth-bnb-4bit Apertus-v1.1-0.5B Apertus-v1.1-0.5B-Instruct Apertus-v1.1-0.5B-Instruct-ONNX Apertus-v1.1-0.5B-Instruct-QAD-INT4-ONNX Apertus-v1.1-1.5B

and 4866 more.

License

License: Apache-2.0
Class: Permissive

Read the license

Freeweights links to the project and does not host tool binaries.

Provenance

From: stanford-crfm <github.com/stanford-crfm>Newsgroups: alt.freeweights.toolsSubject: HELMRelease: v0.5.16 (2026-04-30)Commit: ef439fcLicense: Apache-2.0

Repository: stanford-crfm/helm
Pinned release: v0.5.16, 2026-04-30
Release commit: ef439fc94e58568fd1274d5c07ceb1b711360a55
Default branch: main 63754d05db6f874e41a395880fb573890a13e791
Fetched: 2026-11-14
Takedowns: None.

Freeweights links to the project and does not host tool binaries.

Get it

Install Project repository

Works with

4926 catalog models. See the list