K2-Horizon-3.7B

K2-Horizon-3.7B#

Training Code - Evaluation Code - All Training Data

K2-Horizon-3.7B is the small dense member of the K2-Horizon family: a 3.7B-core decoder-only model with a 512K context window.

Image: K2-Horizon-3.7B benchmark results

K2-Horizon-3.7B Highlights#

• Strong small-model baseline. A dense model evaluated on the same agentic, coding, and reasoning benchmarks as the rest of the family.
• 512K context. Native 524,288-token context from the midtraining stages onward.
• Intermediate checkpoints. Intermediate checkpoints are released so capability changes can be studied across training rather than at a single checkpoint.
• Fully open. Training data and recipe, training code, and evaluation resources are public.

Benchmark Results#

The chart at the top of this card shows K2-Horizon-3.7B against selected reference models. The table below lists every comparison model used in the figure.

Full Results#

| | Open-weight dense models

| | K2-Horizon-3.7B | Qwen3.5-4B | G9v3-3B | Granite 4.2-3B | Nemotron 3 Nano-4B

| # Params | 3.7B | 4B | 3B | 3B | 4B

| # Activated params | 3.7B | 4B | 3B | 3B | 4B

| Architecture | Dense | Dense | Dense | Dense | Dense

| Math

| HMMT Feb 2026

Competition mathematics

| 75.2 | 61.6 | 34.1 | 57.2 | 34.7

| Coding

| SWE-bench Verified

Software engineering

| 65.6 | 41.2 | 16.4 | 32.2 | 1.8

| Scientific Reasoning

| GPQA Diamond

Graduate-level science QA

| 68.9 | 77.1 | 43.8 | 55.9 | 51.3

| HLE

Expert-level reasoning

| 14.0 | 9.9 | 4.5 | 6.6 | 4.9

| Coding

| SciCode

Scientific coding

| 25.8 | 16.1 | 17.7 | 24.9 | 16.4

| Terminal-Bench 2.1

Agentic terminal use

| 39.7 | 25.8 | 6.0 | 13.9 | 3.7

| Agents

| tau3-Banking

Agentic tool use

| 17.7 | 6.8 | — | 5.6 | —

| BFCL v4

Function calling

| 64.6 | 55.7 | 47.9 | 50.8 | 36.8

Scores in %. Baseline protocols may differ;

Anti-doom-loop training#

Small reasoning models occasionally fall into degenerate repetition ("doom loops") on long generations and keep generating until they hit the output limit. After post-training, the 3.7B and 7B models receive a short anti-doom-loop stage: Final Token Preference Optimization (FTPO), a LoRA preference update on pairs mined from the model's own looping outputs, merged back into the weights. Runaway generations are almost entirely removed, standard benchmarks stay flat or improve slightly, and the largest gains are on agentic tasks, where a single runaway call can stall a whole episode.

Output length. On single-turn benchmarks mean output length falls while the median is essentially unchanged, because what FTPO removes is the runaway tail. For example, IFEval mean tokens drop from 8.7k to 2.3k (3.7B) and 5.2k to 3.2k (7B), and LiveCodeBench drops from 26.8k to 16.8k and 19.9k to 15.6k. The share of samples that hit the output limit falls from up to 4% (HLE) to under 0.3% on most benchmarks.

Settings: accuracy in %, "before" is the post-trained checkpoint the anti-doom stage starts from and "after" is the released checkpoint, both evaluated with identical settings. Output budgets are 500k tokens for AIME, HMMT Feb 2025, IFEval, IFBench, LiveCodeBench and OJBench, and 256k for HMMT Feb 2026, GPQA Diamond and HLE. GPQA Diamond uses top_p=1 and 5 repeats; SciCode uses 3 repeats. SWE-bench Verified (mini-swe-agent) and Terminal-Bench 2.1 (terminus-2, 3 repeats) run at 256k context; BFCL v4 runs at temperature 0.01.

Quickstart#

Serving#

vLLM, recipe at recipes.vllm.ai/IFM:

vllm serve IFM/K2-Horizon-3.7B \
  --trust-remote-code \
  --dtype bfloat16 \
  --tensor-parallel-size 1 \
  --reasoning-parser k2_horizon \
  --enable-auto-tool-choice \
  --tool-call-parser k2_horizon

Use an exact branch name from the inventory with vLLM's --revision option. For example, --revision pretrain_1100000 selects the final checkpoint of Pretraining, at step 1,100,000.

SGLang, this is the recipe validated in the SGLang K2 Horizon cookbook:

sglang serve \
  --model-path IFM/K2-Horizon-3.7B \
  --revision fe504ef19c7efa3120a9352bcfa180aa133946de \
  --tp 1 \
  --dtype bfloat16 \
  --attention-backend fa3 \
  --reasoning-parser k2_horizon \
  --host 0.0.0.0 \
  --port 30000

API Usage#

Tip. Recommended settings: reasoning_effort="high", temperature=1.0, top_p=0.95, and at least 32,768 output tokens. Reasoning depth is selected per request through chat_template_kwargs. Thinking is returned in reasoning_content and the answer in content.

from openai import OpenAI

client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY")
response = client.chat.completions.create(
    model="IFM/K2-Horizon-3.7B",
    messages=[{"role": "user", "content": "Explain the result step by step."}],
    temperature=1.0,
    top_p=0.95,
    max_tokens=32768,
    extra_body={"chat_template_kwargs": {"reasoning_effort": "high", "tool_call_format": "xml"}},
)
message = response.choices[0].message
print("Reasoning:", getattr(message, "reasoning_content", None))
print("Answer:", message.content)

Our model supports multiple tool calls formats, which can be changed with chat_template_kwargs. The supported values are json, xml, and xml_typed . The default is xml. Keep --tool-call-parser k2_horizon enabled to parse the selected format.

Transformers#

Validated with Transformers 5.15.0, PyTorch 2.13.0, Safetensors 0.8.0.

from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "IFM/K2-Horizon-3.7B"
tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    model_id, device_map="auto", dtype="bfloat16", low_cpu_mem_usage=True, trust_remote_code=True
)

inputs = tokenizer("Explain why long-context evaluation is difficult.", return_tensors="pt").to(model.device)
inputs.pop("token_type_ids", None)
outputs = model.generate(**inputs, max_new_tokens=32768, temperature=1.0, top_p=0.95, do_sample=True)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Training Overview#

The table below lists the training stages in order and the purpose of each stage.

Training steps are counted within each stage or phase. Token budgets cover only the additional training in that stage or phase. For example, the 50B tokens listed for SFT Phase 2 are additional to the 219B tokens in Phase 1, bringing the cumulative budget to 269B tokens by the end of Phase 2. Here, B and T denote billion and trillion tokens, respectively.

Each stage or phase continues from the final checkpoint of the preceding stage or phase.

Some stages, such as SFT, have multiple phases with slight changes to the data mix while retaining the same overall purpose. During RL, training branches into multiple expert models, which are then merged, as described below.

Release Artifacts#

The tables below list the release artifacts for K2-Horizon-3.7B, their availability, and the expected release dates for remaining items.

Last updated: 2026-10-01

Status:

• Available — fully released for the scope listed;
• Partial — some items are available, with remaining items listed in the notes;
• In Progress — being prepared for release but not yet available.

Artifact Index#

Checkpoint Inventory#

Model repository: IFM/K2-Horizon-3.7B

Branch names below refer to this repository. Patterns containing * group branches by training stage or phase. The * is a placeholder for a training-step number, not a literal branch name. Intermediate checkpoint groups exclude the final checkpoint listed separately; a pattern does not imply that a checkpoint is available at every step.

For example, sft_1_10000 is the checkpoint saved at training step 11,000 within SFT Phase 1, and is the final checkpoint of that phase. The numeric suffix is the step within the named stage or phase, not the cumulative step across all training. Thus, sft_2_2500 refers to step 2,500 within SFT Phase 2.

For a partially released group, the available checkpoints and the remaining checkpoints are listed in the notes.

Important. Note:

• Released K2-Horizon Hugging Face checkpoints (e.g. Huggingface) can be used for inference, evaluation, and downstream fine-tuning (including SFT).

Training behavior in the bundled Hugging Face implementation may differ from native xLLM, including the auxiliary load-balancing loss. To continue the original pretraining with xLLM's training behavior, use the native xLLM checkpoint and xLLM runtime.

Best Practices#

1. Reasoning effort: always high. All reported results use high reasoning effort. Pass {"chat_template_kwargs": {"reasoning_effort": "high"}} on every request; medium and low trade accuracy for speed and are not recommended for evaluation. medium and low effort settings are not recommended except for research on reasoning efforts.
2. Sampling parameters. temperature=1.0, top_p=0.95.
3. Output length. Allow at least 32,768 output tokens so reasoning is never cut off. Truncated reasoning is a failed response, not a shorter one.
4. Serving. Use the validated SGLang recipe above: BF16, TP=1, FlashAttention-3. Full recipes for every K2-Horizon size, with measured H200 latency and throughput, are in the SGLang cookbook.
5. Parsers. Enable the k2_horizon reasoning parser for chat, and add the k2_horizon tool-call parser for agent use. Leave both off for plain completion-style generation.
6. Revisions. Pin a revision tag when reproducibility matters. main is the default checkpoint; base_final and the mid_*_final tags identify training stages.

Citation#

@misc{k2horizon2026,
  title  = {Introducing K2 Horizon: Frontier Performance, Radically Open},
  author = {{IFM Team}},
  year   = {2026},
  url    = {https://ifm.ai/blog/k2/},
}