Trinity-Large-TrueBase

Image: Arcee Trinity Large

Trinity-Large-TrueBase#

Introduction#

Trinity-Large-TrueBase is a base pretraining checkpoint from Arcee AI's Trinity Large training run. It is a 398B-parameter sparse Mixture-of-Experts (MoE) model with approximately 13B active parameters per token. The checkpoint was captured after 10 trillion tokens of pretraining, prior to learning-rate annealing and before any instruction tuning or reinforcement learning.

This checkpoint is intended for research, probing, ablation studies, and downstream fine-tuning and comes without any pre-baked alignment, instruction formatting, or preference optimization.

More details on the training of Trinity Large are available in the technical report.

Model Variants#

The Trinity Large family consists of three checkpoints from the same training run:

• Trinity-Large-TrueBase (this release): 10T-token pre-anneal checkpoint with no instruction data
• Trinity-Large-Thinking: Reasoning-optimized, agentic post-training with extended chain-of-thought
• Trinity-Large-Base: Full 17T-token pretrained foundation model with mid-training anneals
• Trinity-Large-Preview: Lightly post-trained, chat-ready model undergoing active RL

Architecture#

Trinity-Large-TrueBase uses a sparse MoE configuration designed to maximize efficiency while maintaining large-scale capacity.

Note: Extended context support (e.g., 512k) was introduced after this checkpoint and is not available in TrueBase.

Benchmark Results#

Training Configuration#

Pretraining#

• Training tokens: 10 trillion
• Checkpoint type: Pre-anneal
• Instruction data: None
• RLHF or post-training: None

This checkpoint branches from the main Trinity Large run at the 10T-token mark, prior to learning-rate decay or post-training phases.

Optimizers#

Optimizer learning rates after WSD warm-up:

• Adam learning rate: 2e-4
• Muon learning rate: 8e-4

Muon was used to support larger critical batch sizes in a highly sparse MoE regime.

Infrastructure#

• Hardware: 2,048 NVIDIA B300 GPUs
• Parallelism: HSDP + Expert Parallelism
• Compute partner: Prime Intellect
• Data partner: Datology

Image: Powered by Datology

Image: Powered by Datology

Intended Use#

• Studying emergent behavior from large-scale pretraining
• Sparse MoE routing and load-balancing research
• Interpretability, probing, and ablation studies
• Domain-specific fine-tuning from a clean base
• Academic and industrial foundation model research

Rationale for Release#

Most base model releases include instruction data, annealed training dynamics, or early alignment stages. Trinity-Large-TrueBase excludes these, providing an opportunity to study what large-scale models learn from pretraining data alone. This checkpoint is intended as a foundation for research rather than as a finished conversational assistant.

Known Limitations#

• Not aligned for safety, helpfulness, or conversational tone
• Requires substantial compute and expertise to fine-tune
• May exhibit raw or unstable behaviors typical of unaligned models
• No extended-context tuning beyond the 8K pretraining window

License#

Trinity-Large-TrueBase is released under the OpenMDW License, version 1.1 (OpenMDW-1.1).