K2-Horizon-375B-A23B is the flagship of the K2-Horizon family: a sparse Mixture-of-Experts model that stores 375B parameters and runs 23B per token, with a 512K context window. We have released the checkpoints, along with the data and the training code.
Frontier-class agentic performance. On agentic tool use, terminal, and long-horizon workflow benchmarks it matches or beats open-weight MoE models up to 2.6× its size and is competitive with closed frontier models (see Benchmark Results).
512K context. Native 524,288-token context from the midtraining stages onward.
Intermediate checkpoints. Intermediate checkpoints are released so capability changes can be studied across training rather than at a single checkpoint.
Fully open. Training data/recipe and the training code are public.
Scores in %. SWE-Atlas-QnA and SWE Bench Pro: strict = no internet. BrowseComp: different models use different harness, we use the Discard-all@95k context length proposed in DeepSeek-V3.2 technical report. WildClawBench: we use a subset of the English text-only-modality tasks. Apex-Agents: we use a subset of text-only-modality tasks. GDPVal-AA is the Elo rating.
Use an exact branch name from the inventory with vLLM's --revision option. For example, --revision pretrain_ph1_211000 selects the final checkpoint of Pretraining Phase 1, at step 211,000.
Tip. Recommended settings: reasoning_effort="high", temperature=1.0, top_p=0.95. Reasoning depth is selected per request through chat_template_kwargs. Thinking is returned in reasoning_content and the answer in content.
from openai import OpenAI
client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY")
response = client.chat.completions.create(
model="IFM/K2-Horizon-375B-A23B",
messages=[{"role": "user", "content": "Explain the result step by step."}],
temperature=1.0,
top_p=0.95,
max_tokens=32768,
extra_body={"chat_template_kwargs": {"reasoning_effort": "high", "tool_call_format": "xml"}},
)
message = response.choices[0].message
print("Reasoning:", getattr(message, "reasoning_content", None))
print("Answer:", message.content)
Our model supports multiple tool-call formats, which can be changed with chat_template_kwargs. The supported values are json, xml, and xml_typed. The default is xml. Keep --tool-call-parser k2_horizon enabled to parse the selected format.
The table below lists the training stages in order and the purpose of each stage.
Training steps are counted within each stage or phase. Token budgets cover only the additional training in that stage or phase. For example, the 8T tokens listed for Pretraining Phase 2 are additional to the 7T tokens in Phase 1, bringing the cumulative budget to 15T tokens by the end of Phase 2. Here, B and T denote billion and trillion tokens, respectively.
Each stage or phase continues from the final checkpoint of the preceding stage or phase. During RL, training branches into five expert models, which are then merged, as described below.
Some stages, such as SFT, have multiple phases with slight changes to the data mix while retaining the same overall purpose.
Training stage
Training steps
Training tokens
Sequence length
Purpose
Pretraining — Phase 1
211000
7.08T
8K
Pretraining.
Pretraining — Phase 2
240000
8.05T
8K
Continued pretraining from Phase 1, with newer and better data.
Midtraining — Stage 1
32500
1.09T
32K
Context extension.
Midtraining — Stage 2
15000
503B
128K
Context extension.
Midtraining — Stage 3
3500
117B
512K
Context extension.
Midtraining — Stage 4
6000
201B
512K
Continued context extension from Stage 3, with the data mix shifted toward agentic and reasoning SFT data.
RL
-
-
512K
We trained five expert models from the final checkpoint of Midtraining Stage 4: knowledge work, IF, search, tool use, and reasoning. We then merged the expert models.
SFT — Phase 1
2400
81B
512K
SFT for better domain coverage, starting from the merged RL checkpoint.
SFT — Phase 2
6000
201B
512K
Continued from Phase 1 with nearly the same data mix.
SFT — Phase 3
1500
50B
512K
SFT on a high-quality subset of the data used in Phases 1 and 2, with learning rate decay.
RL training steps and token counts are not reported here, so a cumulative token total that includes RL is not provided.
Branch names below refer to this repository. Patterns containing * group branches by training stage or phase. The * is a placeholder for a training-step number, not a literal branch name. Intermediate checkpoint groups exclude the final checkpoint listed separately; a pattern does not imply that a checkpoint is available at every step.
For example, pretrain_ph1_211000 is the checkpoint saved at training step 211,000 within Pretraining Phase 1, and is the final checkpoint of that phase. The numeric suffix is the step within the named stage or phase, not the cumulative step across all training. Thus, pretrain_ph2_240000 refers to step 240,000 within Pretraining Phase 2.
For a partially released group, the available checkpoints and the remaining checkpoints are listed in the notes.
Checkpoint
Branch / repository
Status
Remaining items / expected availability
Pretrain Phase 1 Intermediate Checkpoints
pretrain_ph1_*
Available
N/A
Pretrain Phase 1 Final Checkpoint
pretrain_ph1_211000
Available
N/A
Pretrain Phase 2 Intermediate Checkpoints
pretrain_ph2_*
Available
N/A
Pretrain Phase 2 Final Checkpoint
pretrain_ph2_240000
Available
N/A
Midtrain Stage 1 Intermediate Checkpoints
mid_1_*
Available
N/A
Midtrain Stage 1 Final Checkpoint
mid_1_32500
Available
N/A
Midtrain Stage 2 Intermediate Checkpoints
mid_2_*
Available
N/A
Midtrain Stage 2 Final Checkpoint
mid_2_15000
Available
N/A
Midtrain Stage 3 Intermediate Checkpoints
mid_3_*
Available
N/A
Midtrain Stage 3 Final Checkpoint
mid_3_3500
Available
N/A
Midtrain Stage 4 Intermediate Checkpoints
mid_4_*
Available
N/A
Midtrain Stage 4 Final Checkpoint
mid_4_6000
Available
N/A
RL Knowledge Work Expert Checkpoint
rl_knowledge_work
Available
N/A
RL IF Expert Checkpoint
rl_if
Available
N/A
RL Search Expert Checkpoint
rl_search
Available
N/A
RL Tool Use Expert Checkpoint
rl_tool-use
Available
N/A
RL Reasoning Expert Checkpoint
rl_reasoning
Available
N/A
RL Merged Final Checkpoint
rl_merged
Available
N/A
SFT Phase 1 Intermediate Checkpoints
sft_1_*
Available
N/A
SFT Phase 1 Final Checkpoint
sft_1_2400
Available
N/A
SFT Phase 2 Intermediate Checkpoints
sft_2_*
Available
N/A
SFT Phase 2 Final Checkpoint
sft_2_6000
Available
N/A
SFT Phase 3 Intermediate Checkpoints
sft_3_*
Available
N/A
SFT Phase 3 Final Checkpoint
sft_3_1500
Available
N/A
Important. Note:
Released K2-Horizon Hugging Face checkpoints (e.g. Huggingface) can be used for inference, evaluation, and downstream fine-tuning (including SFT).
Training behavior in the bundled Hugging Face implementation may differ from native xLLM, including the auxiliary load-balancing loss. To continue the original pretraining with xLLM's training behavior, use the native xLLM checkpoint and xLLM runtime.
Reasoning effort: always high. All reported results use high reasoning effort. Pass {"chat_template_kwargs": {"reasoning_effort": "high"}} on every request. medium and low effort settings are not recommended except for research on reasoning efforts.
Sampling parameters.temperature=1.0, top_p=0.95.
Serving. Use the validated SGLang recipe above: BF16, TP=8 on one 8× H200 node, FlashAttention-3, with multithreaded weight loading disabled. Full recipes for every K2-Horizon size, with measured H200 latency and throughput, are in the SGLang cookbook and the vLLM recipe.
Parsers. Enable the k2_horizon reasoning parser for chat, and add the k2_horizon tool-call parser for agent use. Leave both off for plain completion-style generation.
import os
os.environ["HF_ENDPOINT"] = "https://freeweights.allternit.com"
from transformers import AutoModelForCausalLM, AutoTokenizer
tok = AutoTokenizer.from_pretrained("IFM/K2-Horizon-375B-A23B", revision="bb3bfb4b532488c05e84a090301f8259c68b319c")
model = AutoModelForCausalLM.from_pretrained("IFM/K2-Horizon-375B-A23B", revision="bb3bfb4b532488c05e84a090301f8259c68b319c", device_map="auto")
