From: michaelfeil <github.com/michaelfeil>Newsgroups: alt.freeweights.toolsSubject:

Infinity

Release: v3.0.0 (2026-10-26)Commit: 09acbe5License: MIT

Infinity is a high-throughput, low-latency serving engine for text-embeddings, reranking models, clip, clap and colpali

retrievalv3.0.0releaseInfinityC++MIT

FactsLicenseMITLanguageC++KindretrievalRelease2026-10-26PlatformsLinux, macOS, WindowsLinks to the projectpermissive

$ docker run --rm --gpus all -p 8000:8000 -v infinity-cache:/data \
$   ghcr.io/michaelfeil/infinity_v3:nvidia-3.0.0 \
$   --model-id sentence-transformers/all-MiniLM-L6-v2 \
$   --torch-device cuda --dtype bfloat16 \
$   --hostname 0.0.0.0 --port 8000

Overview Install Use with Freeweights License Provenance

Overview

Serves embedding and reranker models behind an OpenAI-compatible API.

Kind: retrieval
License: MIT
Language: C++
Platforms: Linux, macOS, Windows
Latest release: v3.0.0 (2026-10-26)
Repository: michaelfeil/infinity

Topicsbert-embeddingsllmtext-embeddings

Freeweights links to the project and does not host tool binaries.

Install

$ docker run --rm --gpus all -p 8000:8000 -v infinity-cache:/data \
$   ghcr.io/michaelfeil/infinity_v3:nvidia-3.0.0 \
$   --model-id sentence-transformers/all-MiniLM-L6-v2 \
$   --torch-device cuda --dtype bfloat16 \
$   --hostname 0.0.0.0 --port 8000

$ docker run --rm --gpus all -p 8000:8000 -v infinity-cache:/data \
$   ghcr.io/michaelfeil/infinity_v3:nvidia-3.0.0 \
$   --model-id sentence-transformers/all-MiniLM-L6-v2 \
$   --torch-device cuda --dtype bfloat16 \
$   --hostname 0.0.0.0 --port 8000

Checked against the project's README on 2026-11-15. The project's own instructions win when they differ: install docs

Freeweights links to the project and does not host tool binaries.

Use with Freeweights

Use a catalog model

Download the weights from Freeweights first (HF_ENDPOINT or fw.py), then point the tool at the folder.

Catalog models it can run (354)

all-MiniLM-L6-v1 all-MiniLM-L6-v2 all-MiniLM-L6-v2-4bit all-mpnet-base-v1 allenai-specter bert-base-nli-max-tokens bert-base-nli-stsb-mean-tokens bert-large-nli-mean-tokens bge-base-en-v1.5-ONNX bge-large-en-v1.5-ONNX bge-m3-mlx-8bit bge-m3-mlx-fp16 bge-m3-ONNX bge-reasoner-embed-qwen3-8b-0923 bge-small-en-v1.5-GGUF bge-small-zh-v1.5-ONNX BGE-VL-MLLM-S2 BGE-VL-v1.5-mmeb BGE-VL-v1.5-zs BiomedVLP-BioViL-T bitnet-embedding-0.6b bitnet-embedding-270m camembertav2-base camembertav2-base clip-ViT-B-32-multilingual-v1 CLM-v0.1-8B-MLX-8bit CodeRankEmbed core-reranker-2b core-reranker-8b DenseOn DenseOn-ONNX distilbert-base-nli-mean-tokens distilbert-base-nli-stsb-mean-tokens distilbert-multilingual-nli-stsb-quora-ranking distilhubert distiluse-base-multilingual-cased E2Rank-4B E2Rank-8B e5-small-v2-Q8_0-GGUF embeddinggemma-2-4bit embeddinggemma-2-5bit embeddinggemma-2-bf16 embeddinggemma-2-GGUF embeddinggemma-2-mxfp4 embeddinggemma-2-mxfp8 embeddinggemma-2-nvfp4 encodec_48khz ettin-reranker-17m-v1 F2LLM-v2-160M F2LLM-v2-160M-ONNX F2LLM-v2-330M F2LLM-v2-330M-ONNX facebook-dpr-ctx_encoder-multiset-base granite-embedding-30m-english-ONNX granite-embedding-30m-sparse granite-embedding-311m-multilingual-r2-ONNX granite-embedding-reranker-english-r2 granite-vision-3.3-2b-embedding gte-small gte-small-Q8_0-GGUF

and 294 more.

License

License: MIT
Class: Permissive

Read the license

Freeweights links to the project and does not host tool binaries.

Provenance

From: michaelfeil <github.com/michaelfeil>Newsgroups: alt.freeweights.toolsSubject: InfinityRelease: v3.0.0 (2026-10-26)Commit: 09acbe5License: MIT

Repository: michaelfeil/infinity
Pinned release: v3.0.0, 2026-10-26
Release commit: 09acbe5cc421b2981d4385f0dbacc868bccd3ae7
Default branch: main f25b80850a7fc9736310496b64faf9903c01ac97
Fetched: 2026-11-15
Takedowns: None.

Freeweights links to the project and does not host tool binaries.

Get it

Install Project repository

Works with

354 catalog models. See the list