Mage-VL

Mage-VL#

An Efficient Codec-Native Streaming Multimodal Foundation Model

Project Page arXiv GitHub Mage-VL Mage-ViT License: Apache 2.0

Image: Mage-VL

Mage-VL is a codec-native, proactive-streaming multimodal foundation model for image and video understanding, whose visual encoder is trained entirely from scratch at a compact 4B scale. It targets a modern Moravec's paradox of VLMs — strong at complex offline reasoning, yet slow and compute-heavy on simple real-time streaming perception. Instead of decoding video into uniformly-sampled frames and pushing a dense grid of patch tokens through a frozen web-pretrained ViT, Mage-VL follows the structure of modern video codecs: it separates a stream into anchor (I) frames and predicted (P) frames, keeps every anchor patch, and retains only the predicted-frame patches where the codec spends bits — the regions carrying real motion and new detail. This codec-aligned sparsity cuts visual tokens by over 75% while preserving spatio-temporal context, yielding up to 3.5× wall-clock inference speedup over uniform frame sampling.

The system pairs two components:

• Mage-ViT — a from-scratch Codec-ViT visual encoder that allocates tokens by codec-derived spatio-temporal importance, on a shared 16×16 patch grid with 3D rotary position encoding. It is codec-agnostic: the same interface accepts a traditional codec (H.264/AVC, HEVC/H.265) via motion vectors + residual energy, or a neural codec (DCVC-RT) via its learned rate map — no architecture or retraining change.
• Qwen3-4B causal decoder — a Qwen3-4B-Instruct-2507 language backbone (the only pretrained component) that consumes Mage-ViT's variable-length token stream through a lightweight two-layer MLP projector, with a unified interface for images, short/long/ultra-long video, and streaming.

On top of this pair, a System 1 & System 2 dual-process design adds proactive streaming inside a single model: a lightweight cognition gate (System 1) watches each rolling codec window and stays silent on routine content, invoking the full VLM (System 2) only when a response-worthy event completes — no multi-agent pipeline required.

✨ Highlights#

• Codec-native & from scratch. The entire visual stack is trained from scratch — no billion-scale image-text ViT initialization. The bio-inspired predictive-patch mechanism (I/P frames at 16×16) cuts visual-token consumption by over 75% (~1/8 or less of dense frame sampling), letting the model train on videos 8× longer under the same budget.
• Codec-native speedup. Codec tokenization sets a superior accuracy–efficiency frontier — up to 3.5× wall-clock inference speedup over uniform frame sampling at matched accuracy, and the fastest of all compared models on most video benchmarks (single 8×B200 node).
• Data-efficient tokenizer. Trained on only ~100M unlabeled images/videos, Mage-ViT matches or beats frontier encoders trained on billions of image-text pairs (SigLIP2 @ 10B, MoonViT @ 2B) — e.g. 99.33% on CIFAR-10 and 85.69% on ImageNet with 256 tokens, showing web-scale pretraining is not essential for a strong VLM front-end.
• Native-resolution scaling. Variable-resolution pretraining lets Mage-ViT improve monotonically with the token budget (peaking >96.1% Food-101 / >86.3% ImageNet at 676 tokens) where fixed-resolution encoders saturate or degrade.
• Matched-LLM video gains. With the 4B Qwen3 backbone held fixed and only the ViT swapped, Mage-VL improves over Qwen3-VL-4B on every reported video and temporal-grounding benchmark — largest on localization-heavy tasks (+22.5 QVHighlight, +17.1 ActivityNet, +11.0 VSI-Bench, +24.5 VideoEval-Pro).
• Strong for its size. On par with Qwen3-VL-4B on static images, and clearly ahead on video understanding and spatial intelligence (+11.0 VSI-Bench, +53.1 CrossPoint, +5.2 EmbSpatial, +22.5 QVHighlight).
• Proactive streaming, single model. A frozen-backbone cognition gate delivers low-latency, event-gated commentary; it tops TimVal / F1 / ROC-AUC / PR-AUC on SoccerNet streaming and generalizes to real 2026 World Cup broadcasts.

📥 Model#

A single checkpoint, microsoft/Mage-VL, is one unified model that simultaneously provides image & video understanding and the proactive streaming gate — the same weights answer offline image/video questions and drive event-gated commentary. It covers every Mage-VL capability: image understanding, frame-sampled video, traditional H.264/HEVC codec video, neural DCVC-RT codec video, and event-gated streaming. The repository bundles the codec processor, the neural codec package, and the proactive gate weights — no separate understanding, NVC, or streaming checkpoint is required.

We additionally release microsoft/Mage-ViT — the standalone visual encoder from the two-stage, from-scratch ViT pre-training (cluster-discrimination on ~100M unlabeled image/video frames). This is the ViT-pre-trained checkpoint only: it has not gone through the joint VLM training with the language model. Use it as a data-efficient, codec-native visual encoder or as a drop-in ViT for your own multimodal training.

🏗️ Architecture#

Image: Mage-VL proactive streaming framework Proactive streaming framework — Mage-ViT incrementally encodes the continuous stream into codec-native visual features shared by the event gate and the causal decoder. The gate scores each rolling window and stays silent on routine content; when it opens, the decoder emits an event-conditioned response.

Mage-ViT — a from-scratch Codec-ViT visual encoder. On a 16×16 patch grid it keeps all anchor (I) frame patches and only the motion-salient predicted (P) frame patches, cutting visual tokens by over 75% while a shared 3D RoPE preserves spatio-temporal positions.

Mage-VL — a unified model where projected visual tokens and text tokens share one causal Qwen3 decoder. Still images become a single spatial block; videos become temporally-ordered codec windows. In streaming mode, a lightweight cognition gate predicts p_speak = g(h_t) per rolling window (over a recurrent streaming memory kept by an event-preserving feature extractor) and triggers generation when p_speak ≥ τ; the response is decoded by the frozen base model from a local sliding window of the most recent codec segments, and a text query can be injected at any time.

Training — a progressive five-stage supervised curriculum (no preference/RL post-training) that produces one unified model:

1. Multimodal alignment via captions — ~350M dense image captions + 4.2M short-video captions.
2. Instruction tuning + short temporal grounding — ~54M image-instruction samples + 3.4M 30–180s video captions.
3. Temporal-horizon expansion — medium/long video (LLaVA-Video, TimeLens, VideoChat-Flash, Molmo2) with retained image SFT.
4. Codec-native long-context adaptation — 350K long videos as rolling codec windows (up to 384/768 frames).
5. Proactive streaming alignment — a cognition gate fine-tuned on ~3.3M streaming samples with the visual encoder and LLM kept frozen (only the gate is trained).

The five stages together produce a single unified model, Mage-VL, that handles image understanding, offline video reasoning, and proactive streaming — no separate variants are shipped.

Two parts of the pipeline apply an AI4AI (AI-for-AI) paradigm: (1) dense recaptioning runs through an agentic closed loop where a GPT-5 rubric scorer grades captions and a Copilot coding agent co-designs the prompt and harness code (e.g. rendering timestamp overlays) under a human validation gate — improving every downstream OCR/doc/chart/perception benchmark and inspiring SkillOpt-Lite; and (2) Stage-3 uses AI-based diagnostics to decide which video categories, resolutions, and frame counts to train on.

📊 Performance#

Image understanding & spatial intelligence — click to expand

Performance comparison across models. Mage-VL-4B and Qwen3-VL-4B use the same 4B Qwen3 LLM backbone; Phi-4-Multimodal-Instruct (Phi-4-MM, 5.6B) and Phi-4-Reasoning-Vision (Phi-4-R-V, 15B) are reported for reference. – = not run. Bold = best in row.

Video understanding & temporal grounding — click to expand

Bold = best in row.

Proactive streaming (SoccerNet) & online video (OVO-Bench) — click to expand

SoccerNet — response timing (StreamMind protocol, codec-native inputs, zero-tolerance canvas matching). Bold = best in column.

JoyAI's high TriggerAcc comes from predicting silence almost everywhere under SoccerNet's heavy class imbalance, so it collapses on the precision-sensitive metrics; StreamMind is trained in-distribution on SoccerNet, whereas Mage-VL is not.

OVO-Bench — online video understanding (SimpleStream recent-window protocol, 4 frames @ 1 fps; no streaming-specific fine-tuning). Mage-VL sets a new state-of-the-art overall score among streaming architectures. RT-Avg / BT-Avg are the Real-Time Visual Perception / Backward Tracing sub-task averages; Overall is their mean. Bold = best model per column (Human is the reference upper bound).

† HERMES = Qwen2.5-VL-7B + HERMES (4K tokens). Baseline results and table structure follow SimpleStream.

🔬 Key Findings#

Beyond the model, the report distills seven empirical findings for efficient multimodal training:

1. Web-scale pretraining is not essential. A from-scratch backbone on ~100M unlabeled frames matches encoders trained on billions of image-text pairs.
2. Variable-resolution pretraining scales monotonically. Quality keeps improving with the visual-token budget instead of saturating/degrading like fixed-resolution encoders.
3. Codec-native tokenization sets a better accuracy–efficiency frontier — up to 3.5× wall-clock inference speedup over uniform frame sampling.
4. Explicit VideoQA SFT is redundant. Dense video captions + standard image SFT are sufficient for strong zero-shot VideoQA.
5. Motion–spatial synergy. Dynamic video training substantially improves static 2D/3D spatial reasoning.
6. AI4AI data pipeline. Agentic closed-loop feedback + prompt/code co-design systematically lift caption quality and downstream scores (inspired SkillOpt-Lite).
7. Zero-Vision SFT for multimodal RL. Bypassing visual SFT in favor of pure-text reasoning SFT unlocks stronger multimodal RL — a compute-efficient path.

🚀 Quick Start#

A single checkpoint, microsoft/Mage-VL, covers every capability below.

Installation#

For offline Transformers inference:

pip install "transformers>=5.7" accelerate pillow torch torchvision \
  opencv-python codec-video-prep

Codec-based video inference also requires ffmpeg and ffprobe on PATH.

Examples#

Two sample inputs ship with this repository:

Offline inference#

Download inference.py. Offline mode loads the checkpoint with AutoModelForCausalLM.from_pretrained and supports images, frame sampling, and both codec engines:

# image
python inference.py --mode offline --image examples/dog.jpg \
  --question "Describe this image in detail."

The image depicts a dog sitting on a patterned rug. The dog appears to be a medium-sized breed with a thick, fluffy coat. Its fur is primarily white with patches of black and brown. The dog's ears are perked up, and it has a calm and attentive expression. [...]

# video — uniform frame sampling
python inference.py --mode offline --video examples/soccer-broadcast.mp4 \
  --video-backend frames --num-frames 32 \
  --question "Describe this video."

The video opens with a man in a black polo shirt, sporting a short haircut, standing in a stadium. He is holding a yellow microphone with the BBC Sport logo on it. The background reveals a large crowd of spectators. [...]

# video — traditional codec (HEVC/H.264)
python inference.py --mode offline --video examples/soccer-broadcast.mp4 \
  --video-backend codec --codec-engine traditional --num-frames 32 \
  --question "Describe this video."

The video opens with a BBC Sport broadcast, featuring a presenter in a black shirt holding a yellow microphone. The background reveals a packed stadium, with the scoreboard displaying "ENG 1 ARG 2 FT", indicating the final score of the match. [...]

# video — neural codec (DCVC-RT)
python inference.py --mode offline --video examples/soccer-broadcast.mp4 \
  --video-backend codec --codec-engine neural --num-frames 32 \
  --question "Describe this video."

The video opens with a BBC Sport broadcast, featuring a presenter standing in a stadium filled with spectators. The presenter, dressed in a black shirt, holds a yellow BBC Sport microphone and wears a black earpiece. [...]

Online inference#

Online mode talks to an OpenAI-compatible SGLang server. First build and launch the server with the Mage-VL SGLang branch (building it needs protobuf-compiler and a Rust toolchain):

sudo apt-get update && sudo apt-get install -y protobuf-compiler
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs \
  | sh -s -- -y --profile minimal --default-toolchain 1.90.0
source "$HOME/.cargo/env"

git clone -b feat/mage-vl https://github.com/kcz358/sglang
cd sglang
pip install -e 'python[all]'
python -m sglang.launch_server \
  --model-path microsoft/Mage-VL \
  --trust-remote-code

Then send an image or sampled video frames to the running server:

pip install openai

python inference.py --mode online --image examples/dog.jpg \
  --question "Describe this image in detail." \
  --base-url http://localhost:30000/v1

python inference.py --mode online --video examples/soccer-broadcast.mp4 \
  --num-frames 32 \
  --question "Describe this video." \
  --base-url http://localhost:30000/v1

Use --model, --max-new-tokens, and --api-key to override their defaults.

Streaming inference#

streammind_gate.safetensors in this repository holds the event gate. Streaming inference splits a video into non-overlapping segments, stays silent on routine content, and generates a caption only when a response-worthy event is detected. Run it with inference_streaming.py from the GitHub repository:

python inference_streaming.py \
  --video examples/soccer-broadcast.mp4 \
  --video_backend codec \
  --segment_sec 8

[t=0.0-8.0s] gate=silence (p=0.19)
[t=8.0-16.0s] gate=response (p=0.55) -> The video features a live sports broadcast from BBC Sport, set in a large stadium filled with spectators. The broadcast focuses on a football match between England and Argentina, with the score displayed as England 1, Argentina 2. [...]
[t=16.0-24.0s] gate=response (p=0.73) -> The video features a sports broadcast set in a large stadium filled with spectators. Four commentators are gathered around a table with a 'BBC Sport' logo, each holding a yellow microphone. [...]
[t=24.0-30.0s] gate=silence (p=0.31)

The gate is trained on codec inputs, so --video_backend codec is the intended setting. Use --video_backend frames for direct frame sampling. Additional controls include --num_frames, --cur_fps, --max_segments, --max_new_tokens, --gate_threshold, and --attn_impl.

📝 Citation#

@article{yang2026mage,
  title={Mage-VL: An Efficient Codec-Native Streaming Multimodal Foundation Model},
  author={Yang, Senqiao and Zhang, Kaichen and Jia, Zhaoyang and Guo, Jinghao and Shen, Yifei and Zhang, Xinjie and Zhang, Xiaoyi and Wang, Haoqing and Li, Xiao and Zhang, Peng and others},
  journal={arXiv preprint arXiv:2607.24904},
  year={2026}
}

📄 License#

Mage-VL is released under the Apache-2.0 License.