Xenon-26B-A4B

Built with Axolotl

Xenon-26B-A4B#

// DIVINE EXECUTION // MOE ROUTING PERFECTION

Gemma-4-26B-A4B LoRA SFT Agentic Coder Strategic MoE Routing

01 The Vessel Refined

The base Gemma-4-26B-A4B architecture possesses immense latent power, but lacked the strict directive to execute agentic workflows reliably. Base models hesitate, hallucinate tool calls, and forget standard library imports. Xenon-26B-A4B resolves this through precise, surgical LoRA adaptation.

Rather than full-parameter fine-tuning, which risks catastrophic forgetting, this model targets the Mixture-of-Experts routing layers. By training the router and expert feed-forward networks, the model learns to dynamically activate specialized sub-networks based on the language paradigm, creating a flawless execution loop.

02 Strategic Dataset Architecture

The defining feature of this model is its training data. Instead of diluting the model's expertise with an even split across 12 languages, we engineered a 3,186-row dataset using Strategic Language Weighting. This guarantees the MoE router naturally learns to route specific language tokens to the most capable base experts.

Access the Strategic Dataset: el4/xenon-mixed-agentic-datasetv1 →

Tier 1: The Core 60%

LanguagesPython, Bash, Zsh

Sources90% Fable-5 / 5% GLM / 5% Sol

Fable-5 dominates this tier, providing the absolute best Python/Bash tool-use traces available. The router learns to send these tokens to the supreme execution experts.

Tier 2: Web & Scripting 25%

LanguagesTypeScript, JS, Ruby

Sources50% Fable-5 / 30% Sol / 20% GLM

GPT-5.6-Sol takes precedence here, offering superior full-stack and frontend agentic workflows. JS/TS tokens are routed to the full-stack specialists.

Tier 3: Systems & Enterprise 15%

LanguagesRust, Go, C++, C#, Java

Sources40% GLM / 40% Sol / 20% Fable-5

GLM-5.2 and Sol handle strict memory safety, algorithmic logic, and enterprise patterns. Systems tokens are routed to the logic and structural experts.

03 The Surgical Strike

Standard SFT often drowns models in 50,000+ rows of noise, causing overfitting and loss of general reasoning. This model was trained for exactly 1 Epoch on highly distilled data. 1 epoch is the gold standard for high-signal distillation; it forces the model to internalize the routing and tool-formatting patterns without memorizing the exact traces.

To prevent the MoE router from collapsing—sending all tokens to a single expert—we enforced an auxiliary loss penalty of 0.01. The LoRA adapters were explicitly targeted at the experts and router modules, ensuring the base 26B parameters remained frozen and pristine.

✦ Axolotl Training Configuration

base_model: google/gemma-4-26b-A4B-it model_type: gemma4_moe trust_remote_code: true adapter: lora lora_r: 16 lora_alpha: 32 lora_dropout: 0.0 # MUST be 0.0 for lora_target_parameters lora_target_modules: 'model.language_model.layers.[\d]+.(_checkpoint_wrapped_module.)?(mlp|self_attn).(up|down|gate|q|k|v|o)_proj' lora_target_parameters:

• experts.gate_up_proj
• experts.down_proj

aux_loss_alpha: 0.01 learning_rate: 1.0e-4 lr_scheduler: cosine warmup_steps: 20 max_steps: 250 # <--- SANITY CHECK FIRST! Change to 250 later. gradient_accumulation_steps: 16 micro_batch_size: 1 max_seq_length: 8192 sequence_len: 8192 bf16: true gradient_checkpointing: true flash_attention: false attn_implementation: sdpa datasets:

• path: mixed_coding_traces.jsonl type: chat_template

save_steps: 50 logging_steps: 5 output_dir: ./outputs/gemma4-26b-mixed

04 Evaluation & Invocation

This model expects strict ChatML formatting. It is designed to operate as an autonomous agent within harnesses like OpenCode, OpenHands, or Aider. It will not merely write code; it will execute workflows.

{"role": "system", "content": "You are an autonomous coding agent. Reproduce failures, inspect files, make the smallest coherent fix, and verify."} {"role": "user", "content": "The deployment script is mangling spaces in arguments. Fix it."} {"role": "assistant", "content": "I will inspect the repository state and reproduce the failure.", "tool_calls": [{"id": "call_1", "type": "function", "function": {"name": "Bash", "arguments": "{\"command\": \"ls -la\"}"}}]}

  BENCHMARKS COMING SOON // HumanEval, SWE-bench, & Terminal-Bench evaluation currently in progress.

May the uncomfortable-olive-cockroach rest in peace 🙏