NVIDIA-Nemotron-3.5-Super-VL-120B-A12B

Read our How to Run Nemotron 3.5 Super VL Guide!#

See Unsloth Dynamic 3.0 GGUFs for our quantization benchmarks.

https://github.com/unslothai/unsloth/ https://discord.gg/unsloth https://unsloth.ai/docs/models/nemotron-3.5-super-vl

NVIDIA-Nemotron-3.5-Super-VL-120B-A12B#

Pre-Training Datasets Homepage Discord

License

https://unsloth.ai/docs/models/nemotron-3.5-super-vl

Description#

NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-BF16 is the full-precision release of NVIDIA Nemotron 3.5 Super VL, a vision-language model for reasoning, coding, tool use, document understanding, and video understanding. The BF16 weights support customization, domain adaptation, and full-precision evaluation. For optimized deployment, see the NVFP4 release.

The model combines interleaved Mamba-2, Mixture-of-Experts (MoE), and attention layers with a C-RADIO vision encoder and a vision-language connector. It has 120B total parameters and 12B active parameters and supports context lengths up to 1M tokens. Multi-Token Prediction (MTP) enables native speculative decoding.

This model is ready for commercial or non-commercial use.

Model Summary#

Model Overview#

Model Developer: NVIDIA Corporation

What is Nemotron?#

NVIDIA Nemotron™ is a family of open models with open weights, training data, and recipes, delivering leading efficiency and accuracy for building specialized AI agents.

License/Terms of Use#

Governing Download Terms: Use of this model is governed by the OpenMDW License Agreement, version 1.1.

Benchmarks#

We evaluated our model on the following benchmarks:

We evaluated Nemotron 3.5 Super on a comprehensive suite of benchmarks spanning agentic capabilities, reasoning and knowledge, multimodal, conversational ability and instruction following, long-context understanding, and multilingual performance. All evaluation results for Nemotron 3.5 Super and baselines were collected via NeMo Gym. For reproducibility purposes, more details on the evaluation settings can be found in the NeMo Gym recipes folder and the corresponding reproducibility tutorial. The published recipes specify the benchmark-specific containers, prompts, inference parameters, parser configurations, and scoring settings used to produce the results.

There were a few benchmarks that were not yet onboarded in our open source tools, and for these we used either their official open source implementation or an internal scaffolding that we plan to open source in the future: BabyVision, CharXiv-RQ, MathVision, MMLongBench-Doc, OCR-Reasoning, OCRBench-v2, V*, Video-MME-v2.

Agentic Coding Benchmarks#

Additional harness-level coding-agent results for SWE-Bench Pro and Terminal-Bench 2.1 are shown below.

Image: image

Deployment Geography: Global#

Use Case#

This model is intended for developers who want to customize a post-trained vision-language model, adapt it to domain-specific data, develop quantized variants, or run full-precision inference and evaluation.

Release Date#

Hugging Face - 10/15/2026 via Hugging Face

Reference(s)#

• NVIDIA Nemotron 3.5 Super VL and Lightning Technical Report

Model Architecture#

• Architecture Type: Mamba-2–Transformer hybrid Mixture of Experts (MoE) with Multi-Token Prediction (MTP)
• Network Architecture: Nemotron Hybrid MoE with 512 routed experts per MoE layer, a C-RADIO vision encoder, and a vision-language connector
• Number of Model Parameters: 120B total / 12B active

Model Design#

The model combines sparse expert computation, attention, and Mamba-2 layers to support long-context processing. The vision encoder and connector turn visual inputs into representations processed by the language backbone. A shared MTP head predicts future tokens recursively for speculative decoding.

The model supports up to 1M tokens of context. The prompt, visual inputs, conversation history, tool results, and generated response share this context window. Individual deployment recipes may use a smaller context window to accommodate the available GPU memory and concurrency.

Training Methodology#

Stage 1: Foundation and Multi-Token Prediction

The language backbone is initialized from Nemotron 3 Super Base. A shared MTP head is trained for 100B tokens while the backbone remains frozen, followed by 1T tokens of distillation from Nemotron 3 Ultra Base.

Stage 2: Continued Pre-Training and Mid-Training

The model undergoes 100B tokens of Indic continued pre-training, a 35B-token extension to 1M context, 67B tokens of software-engineering mid-training at 256K context, and a final 9B-token long-context recovery stage. Software-engineering data includes repositories, issue–pull-request pairs, commit diffs, build and CI logs, and agent trajectories.

Stage 3: Vision-Language Training

A C-RADIO vision encoder and connector extend the language backbone to visual inputs, including images, document pages, and video frames.

Stage 4: Supervised Fine-Tuning

A two-stage supervised fine-tuning pipeline jointly trains the language and vision components on vision, reasoning, code, tools, chat, safety, and long-context data. Training uses packed sequences of up to 524,288 tokens.

Stage 5: Reinforcement Learning and Distillation

Reinforcement learning with verifiable rewards, reinforcement learning from human feedback, and specialized teacher training improve reasoning, agentic, chat, and vision capabilities. Multi-teacher on-policy distillation consolidates these capabilities into a single model.

Stage 6: MTP Boosting

The MTP head is further trained against the frozen post-trained backbone to improve speculative-decoding acceptance.

Input#

• Input Type(s): Text, images, and video
• Input Format(s): Text strings, images, and sampled video frames
  • Video: mp4, up to 2 minutes.
  • Image: Red, Green, Blue (RGB) (jpeg, png)
• Input Parameters:
  • Video: Three-Dimensional (3D)
  • Image: Two-Dimensional (2D)
  • Text: One-Dimensional (1D)
• Other Properties Related to Input: Maximum context length up to 1M tokens. Documents can be supplied as page images with accompanying text.

Output#

• Output Type(s): Text
• Output Format: String
• Output Parameters: One-dimensional sequences
• Other Properties Related to Output: Maximum context length up to 1M tokens.

Our AI models are designed and optimized to run on NVIDIA GPU-accelerated systems. By leveraging NVIDIA's hardware (e.g. GPU cores) and software frameworks (e.g., CUDA libraries), the model achieves faster training and inference times compared to CPU-only solutions.

Software Integration#

• Runtime Engine(s): vLLM; SGLang
• Hardware:
  • NVIDIA Blackwell
  • NVIDIA Hopper
• Operating System(s): Linux

The integration of foundation and fine-tuned models into AI systems requires additional testing using use-case-specific data to ensure safe and effective deployment. Following the V-model methodology, iterative testing and validation at both unit and system levels are essential to mitigate risks, meet technical and functional requirements, and ensure compliance with safety and ethical standards before deployment.

Training, Testing, and Evaluation Datasets#

Training#

Data Modality: Text, images, and video Foundation Training Data Size: More than 20 trillion tokens Image Training Data Size: 1 Million to 1 Billion Images Text Training Data Size: More than 10 Trillion Tokens Video Training Data Size: 10,000 to 1 Million Hours Foundation Data Collection Period: 2013 to December 2025 Data Collection Method: Hybrid: automated, manually-collected, and synthetic Labeling Method: Hybrid: automated, manually-labeled, and synthetic

The foundation corpus contains curated and synthetic text covering English, 19 other spoken languages, and 43 programming languages. Sources include webpages, dialogue, articles, code, and other written materials across math, science, legal, finance, and general knowledge domains.

The post-training corpus includes reasoning, coding, tool use, instruction following, structured outputs, chat, safety, and long-context tasks, together with image and video understanding. Synthetic trajectories are filtered for structural correctness, repetition, and task validity.

For more information about the datasets used to train this model, please see the Public Summary of Training Content

Detailed dataset information

Base Pre-Training Corpus (Nemotron 3 Foundation)#

The foundation of the model is trained on the Nemotron 3 corpus, comprising the following datasets from the Nemotron Pretraining Datasets collection:

Public Datasets#

Crawled and Scraped from Online Sources by NVIDIA#

The English Common Crawl data was downloaded from the Common Crawl Foundation (see their FAQ for details on their crawling) and includes the snapshots CC-MAIN-2013-20 through CC-MAIN-2025-13. The data was subsequently deduplicated and filtered in various ways used in the Nemotron-CC data pipeline. Additionally, we extracted data for fifteen languages from the following three Common Crawl snapshots: CC-MAIN-2024-51, CC-MAIN-2025-08, CC-MAIN-2025-18. The fifteen languages included were Arabic, Chinese, Danish, Dutch, French, German, Italian, Japanese, Korean, Polish, Portuguese, Russian, Spanish, Swedish, and Thai. As we did not have reliable multilingual model-based quality classifiers available, we applied just heuristic filtering instead—similar to what we did for lower quality English data in the Nemotron-CC pipeline, but selectively removing some filters for some languages that did not work well. Deduplication was done in the same way as for Nemotron-CC.

The GitHub Crawl was collected using the GitHub REST API and the Amazon S3 API. Each crawl was operated in accordance with the rate limits set by its respective source, either GitHub or S3. We collect raw source code and subsequently remove any having a license which does not exist in our permissive-license set.

Private Non-publicly Accessible Datasets of Third Parties#

Private Non-publicly Accessible Datasets by NVIDIA#

NVIDIA-Sourced Synthetic Datasets (Pre-Training)#

NVIDIA-Sourced Synthetic Datasets (Post-Training)#

The publisher's README is longer; it is cut here. Read the rest at the source.