[2025.09.29] InfLLM-V2 paper is released! We can train a sparse attention model with only 5B long-text tokens. 🔥🔥🔥
[2025.09.05] MiniCPM4.1 series are released! This series is a hybrid reasoning model with trainable sparse attention, which can be used in both deep reasoning mode and non-reasoning mode. 🔥🔥🔥
[2025.06.06] MiniCPM4 series are released! This model achieves ultimate efficiency improvements while maintaining optimal performance at the same scale! It can achieve over 5x generation acceleration on typical end-side chips! You can find technical report here.🔥🔥🔥
It is advisable to use temperature=0.9, topp=0.95. And we suggest setting max_output_token to 65,536 tokens.
For math problems, we recommend using "Please reason step by step, and put your final answer within \boxed{}."
And for English multiple-choice questions, we recommend starting with "Answer the following multiple choice question. The last line of your response should be of the following format: 'ANSWER: $LETTER' (without quotes) where LETTER is one of ABCD. Think step by step before answering." And "你回答的最后一行必须是以下格式 '答案：$选项' (不带引号), 其中选项是ABCD之一。请在回答之前一步步思考" for Chinese MCQ.
MiniCPM4.1 adopts sparse attention and speculative decoding to improve the inference efficiency. On RTX 4090, MiniCPM4.1 achieves 3x decoding speed improvement in reasoning.
MiniCPM 4.1 can be used with following frameworks: Huggingface Transformers, SGLang, vLLM, and CPM.cu. For the ultimate inference speed, we highly recommend CPM.cu.
MiniCPM4/MiniCPM4.1 supports both dense attention inference and sparse attention inference modes, where vLLM and SGLang currently only support dense inference mode. If you want to use sparse inference mode, please use Huggingface Transformers and CPM.cu.
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch
torch.manual_seed(0)
path = 'openbmb/MiniCPM4.1-8B'
device = "cuda"
tokenizer = AutoTokenizer.from_pretrained(path)
model = AutoModelForCausalLM.from_pretrained(path, torch_dtype=torch.bfloat16, device_map=device, trust_remote_code=True)
# User can directly use the chat interface
# responds, history = model.chat(tokenizer, "Write an article about Artificial Intelligence.", temperature=0.7, top_p=0.7)
# print(responds)
# User can also use the generate interface
messages = [
{"role": "user", "content": "Write an article about Artificial Intelligence."},
]
prompt_text = tokenizer.apply_chat_template(
messages,
tokenize=False,
add_generation_prompt=True,
)
model_inputs = tokenizer([prompt_text], return_tensors="pt").to(device)
model_outputs = model.generate(
**model_inputs,
max_new_tokens=32768,
top_p=0.95,
temperature=0.6
)
output_token_ids = [
model_outputs[i][len(model_inputs[i]):] for i in range(len(model_inputs['input_ids']))
]
responses = tokenizer.batch_decode(output_token_ids, skip_special_tokens=True)[0]
print(responses)
Inference with Sparse Attention
MiniCPM4.1-8B supports InfLLM v2, a sparse attention mechanism designed for efficient long-sequence inference. It requires the infllmv2_cuda_impl library.
You can install it by running the following command:
These parameters control the behavior of InfLLM v2:
kernel_size (default: 32): The size of semantic kernels.
kernel_stride (default: 16): The stride between adjacent kernels.
init_blocks (default: 1): The number of initial blocks that every query token attends to. This ensures attention to the beginning of the sequence.
block_size (default: 64): The block size for key-value blocks.
window_size (default: 2048): The size of the local sliding window.
topk (default: 64): The specifies that each token computes attention with only the top-k most relevant key-value blocks.
use_nope (default: false): Whether to use the NOPE technique in block selection for improved performance.
dense_len (default: 8192): Since Sparse Attention offers limited benefits for short sequences, the model can use standard (dense) attention for shorter texts. The model will use dense attention for sequences with a token length below dense_len and switch to sparse attention for sequences exceeding this length. Set this to -1 to always use sparse attention regardless of sequence length.
Long Context Extension
MiniCPM4.1 natively supports context lengths of up to 65,536(64k) tokens. For conversations where the total length (including both input and output) significantly exceeds this limit, we recommend using RoPE scaling techniques for effective handling of long texts. We have validated the model's performance on context lengths of up to 131,072 tokens by modifying the LongRoPE factor.
You can apply the LongRoPE factor modification by modifying the model files. Specifically, in the config.json file, adjust the rope_scaling fields.
Start the vLLM inference server with speculative decoding enabled. Make sure to update the model path in the speculative-config to point to your downloaded MiniCPM4_1-8B-Eagle3-bf16 folder:
Also, you can start the inference server by running the following command:
Note: In vLLM's chat API, add_special_tokens is False by default. This means important special tokens—such as the beginning-of-sequence (BOS) token—will not be added automatically. To ensure the input prompt is correctly formatted for the model, you should explicitly set extra_body={"add_special_tokens": True}.
We recommend using CPM.cu for the inference of MiniCPM4 and MiniCPM4.1. CPM.cu is a CUDA inference framework developed by OpenBMB, which integrates efficient sparse, speculative sampling, and quantization techniques, fully leveraging the efficiency advantages of MiniCPM4 and MiniCPM4.1.
You can install CPM.cu by running the following command:
git clone https://github.com/OpenBMB/cpm.cu.git --recursive
cd cpm.cu
python3 setup.py install
MiniCPM4.1 natively supports context lengths of up to 65,536(64k) tokens. To reproduce the long-text acceleration effect in the paper, we recommend using the LongRoPE factors that have been validated. Change the rope_scaling field in the config.json file as the following to enable LongRoPE.
After modification, you can run the following command to reproduce the long-context acceleration effect (the script will automatically download the model weights from HuggingFace)
python3 tests/test_generate.py
You can run the following command to infer with EAGLE3 speculative decoding algorithm.
python3 -m cpmcu.cli \
--model-path $BASE_MODEL_PATH \
--draft-model-path $EAGLE3_DRAFT_MODEL_PATH \
--prompt-text "Write an article about Artificial Intelligence." \
--use-eagle3 true
For more details about CPM.cu, please refer to the repo CPM.cu.
MiniCPM4.1 supports hybrid reasoning mode, which can be used in both deep reasoning mode and non-reasoning mode. To enable hybrid reasoning mode. User can set enable_thinking=True in tokenizer.apply_chat_template to enable hybrid reasoning mode, and set enable_thinking=False to enable non-reasoning mode. Similarly, user can directly add /no_think at the end of the query to enable non-reasoning mode. If not add any special token or add /think at the end of the query, the model will enable reasoning mode.
import os
os.environ["HF_ENDPOINT"] = "https://freeweights.allternit.com"
from transformers import AutoModelForCausalLM, AutoTokenizer
tok = AutoTokenizer.from_pretrained("openbmb/MiniCPM4.1-8B", revision="3a8dfed9c79a45e07dbff95bcd49d792343fa1a3")
model = AutoModelForCausalLM.from_pretrained("openbmb/MiniCPM4.1-8B", revision="3a8dfed9c79a45e07dbff95bcd49d792343fa1a3", device_map="auto")
