Mage-VL-Cowork

Mage-VL-Cowork#

A Full Training Recipe to Learn Hybrid Computer Use from Evolved Harnesses

Project Page Technical Report GitHub Mage-VL-Cowork Mage-VL Mage Collection License: Apache 2.0

Image: Mage-VL-Cowork achieves 66.0% on OSWorld-Verified with 4B parameters

One compact 4B policy that knows when to click — and when to code.

Computer-use agents (CUAs) that confine themselves to pixels pay a heavy tax: a single screenshot consumes thousands of tokens, and re-deriving structured file edits through dozens of mouse clicks discards the fastest, most verifiable channel on the machine. Conversely, giving a frontier model naive access to both GUI and terminal tools does not automatically produce hybrid competence — without an optimized execution scaffold, even frontier teachers exhibit suboptimal channel habits that transfer directly to distilled students. Worse, existing CUA post-training pipelines rely on checkpoints already heavily specialized for GUI grounding and coding.

Mage-VL-Cowork bridges this gap. Trained end-to-end from an intermediate 4B Mage-VL checkpoint through a progressive five-stage specialize-then-unify curriculum, it combines screenshot-grounded GUI interaction (computer_use) and persistent code/terminal execution (bash) inside a single vision-language policy — with no external orchestrator. Upstream of training, we evolve the teacher harness (pruning redundant action routes and equipping domain libraries) and build Mage-CUA-Forge, a workflow-first synthesis engine that generates 293,806 tasks across 6,915 atomic skills while cutting recurring per-task generation cost by 82.6%.

At 4B parameters, Mage-VL-Cowork reaches 66.0% on OSWorld-Verified under a 50-step budget — achieving the best performance among 4B models and demonstrating that an efficient small VLM can excel at complex computer use, pushing the frontier of on-device intelligence. Across training stages, online reinforcement learning adds +10.16 percentage points over Hybrid SFT while simultaneously cutting mean model calls by 13.0% and output tokens by 25.4%.

✨ Highlights#

• Best-in-class 4B hybrid computer use (66.0% OSWorld-Verified). Achieving the highest score at the 4B scale, Mage-VL-Cowork proves that an efficient small VLM can master end-to-end computer use — leading on Operating System tasks at 87.5%, reaching 62.7% on Multi-Application Workflows, and advancing the frontier of on-device intelligence.
• Internalized turn-level channel routing. Without an external multi-agent router, the policy dynamically selects computer_use for visual layouts, canvas editing, and live in-memory application state, and switches to bash for structured programmatic edits (spreadsheets, documents, presentations) and deterministic ground-truth verification.
• Evolved teacher harnesses amplify student distillation (+12.5 pp). Evolving and pruning the teacher scaffold before rollout (removing overlapping tools like xdotool and accessibility trees while pre-installing domain Python libraries) lifts the GPT-5.4 teacher to 92.0% with 29.2% fewer steps, which expands into a +12.5 pp student gain (37.6% vs. 25.1%) at the 4B scale.
• Workflow-first synthesis at −82.6% recurring cost. Mage-CUA-Forge authors reusable, skill-conditioned generation workflows and application-specific preparers once per skill rather than running open-ended agent loops per task — reducing token consumption by 93.3% (1,276K → 85K tokens/task) and expenditure from $0.906 to $0.158 per task.
• Specialize-then-unify five-stage curriculum (+20.44 pp). Starting from an intermediate 4B Mage-VL checkpoint, the model builds standalone visual grounding (30.6% → 56.9% on OSWorld-G; 78.8% → 87.2% on ScreenSpot-v2), GUI execution, and coding/terminal competence before unifying them via Hybrid SFT and asynchronous Online Hybrid RL (45.56% → 55.84% → 66.0%).
• **Stronger and more compact after Online RL. Online GRPO with verifiable task rewards improves task accuracy by +10.16 pp over Hybrid SFT while compressing interaction length from 21.57 to 18.77 model calls (−13.0%) and output tokens from 7,374 to 5,499 per task** (−25.4%).

📥 Model Overview#

A single unified checkpoint, microsoft/Mage-VL-Cowork, handles visual grounding, multi-step GUI manipulation, programmatic file/system operations, and dynamic GUI-to-CLI channel switching.

🏗️ Architecture & Training Recipe#

Image: Mage-VL-Cowork pipeline

Overview of Mage-VL-Cowork: upstream teacher-harness evolution, reusable skill-conditioned task workflows (Mage-CUA-Forge), rubric judging, and student-feedback-driven skill expansion.

1. Evolved Hybrid CUA Harness#

A strong teacher model does not automatically produce clean hybrid supervision. Before large-scale trajectory collection, an authoring agent evolves the external harness $\mathcal{H} = (\mathcal{F}, \mathcal{T}, \mathcal{P}, \mathcal{E})$ — refining the execution scaffold $\mathcal{F}$, tool interfaces $\mathcal{T}$, system prompt $\mathcal{P}$, and pre-installed terminal environment $\mathcal{E}$ (equipping domain packages such as python-pptx, openpyxl, pandas, pypdf, and pillow while removing redundant tools like xdotool and verbose accessibility dumps). We instantiate specialized GUI, Terminal, and Hybrid harnesses to roll out the same synthesized tasks across training stages.

2. Workflow-First Environment Synthesis (Mage-CUA-Forge)#

Instead of spending open-ended agent tokens to construct one task at a time, Mage-CUA-Forge invests authoring effort once per atomic skill to build reusable generation workflows and application-specific preparers. Shared specification branches coordinate input artifact creation, executable environment setup scripts, user instructions, and evaluation rubrics. For web and cross-application tasks, we pair desktop preparers with synthesized InfiniteWeb sites and deterministic, SQLite-backed WebHarbor replicas that reset cleanly before every episode. Across three student-feedback-guided rounds, the pipeline synthesizes 293,806 distinct tasks (38.24% multi-application, 877,608 file references across 278 file categories) and retains 318,837 verified hybrid trajectories (68.19% GUI tool calls, 31.81% terminal tool calls).

3. Five-Stage Specialize-Then-Unify Curriculum#

Rather than learning hybrid routing as a workaround for weak visual grounding, Mage-VL-Cowork builds two strong standalone channels first and then unifies them:

4. Asymmetric Working Memory#

Long-horizon desktop tasks face a tension between context budget and visual state awareness. At step $t$, Mage-VL-Cowork conditions on an asymmetric sliding window:

• Folded long-horizon memory ($1 \dots t-4$): Earlier turns are compressed into single-line imperative Action: summaries placed immediately after the system prompt.
• Active visual window ($t-3, t-2, t-1$): The 3 most recent screenshots and 4 complete assistant responses are retained in full so the model can directly inspect before-and-after UI transitions and recover from no-op clicks.

📊 Performance & Experimental Evidence#

Every evaluation below validates a core pillar of the Mage-VL-Cowork design.

1. Main Desktop Results: OSWorld-Verified (66.0% at 4B)#

Evaluation Protocol: Evaluated on 361 held-out OSWorld-Verified tasks (excluding Google Drive) under a 50-step budget using the MAGEHybridAgent harness (history_n=3, images_to_keep=3, 1920×1080 desktop resolution). Following our rigorous multi-run standard, the headline 66.0% (66.04%) is the two-run average across two independent full-benchmark passes (65.33% on Run 1 and 66.75% on Run 2).

† Evaluated in one run on the same 361 tasks excluding Google Drive.

Key Takeaways:

• Best-in-class at the 4B scale & a leap for on-device intelligence: At just 4B parameters, Mage-VL-Cowork achieves 66.0% on OSWorld-Verified — the highest performance among 4B models — proving that an efficient small VLM can master complex computer use and pushing the frontier of on-device intelligence.
• Strong OS & Multi-Application Workflow execution: Mage-VL-Cowork reaches 87.5% on OS tasks and 62.7% on Multi-Application Workflows, demonstrating how unified GUI + terminal execution turns complex cross-application desktop tasks into compact, reliable trajectories.

1. Stage-Wise Progression & Inference Efficiency (+20.44 pp & −25.4% tokens) — click to expand

Across the downstream curriculum, Mage-VL-Cowork climbs by +20.44 percentage points from GUI SFT to Online Hybrid RL while simultaneously becoming faster and more concise. In Stage 1 (GUI-oriented midtraining), grounding accuracy rises from 30.6% → 56.9% on OSWorld-G (surpassing Qwen3.5-4B at 54.5%) and 78.8% → 87.2% on ScreenSpot-v2 (surpassing Qwen3-VL-4B at 85.3%).

• RL sharpens both accuracy and token economy: Relative to Hybrid SFT, Online Hybrid RL adds +10.16 pp while cutting mean model calls by 13.0% (21.57 → 18.77) and output tokens by 25.4% (7,374 → 5,499).
• 2.25× more token-efficient than GPT-5.4-mini: Mage-VL-Cowork matches GPT-5.4-mini in interaction turns (18.77 vs. 18.24 calls/task) while consuming 55.6% fewer output tokens (5,499 vs. 12,397 tokens/task).

1. Why Evolving the Teacher Harness Matters (+12.5 pp Student Gain) — click to expand

More tools do not automatically make a better agent. Holding the teacher model fixed (GPT-5.4) and evaluating on 672 held-out multi-application tasks, we compare a clean GUI + Coding ($\beta_1$) harness against an Accessibility-augmented GUI + Coding + AX ($\beta_2$) harness, and train a 4B student on each teacher's trajectories:

While a frontier teacher can partially absorb a redundant tool channel by taking extra steps, a compact 4B student is highly sensitive to split supervision. Pruning overlapping action routes upstream yields a +12.5 percentage-point student jump and 21.5% shorter trajectories.

1. Workflow-First Data Synthesis Efficiency (−82.6% Cost) — click to expand

Using the exact same generation model (gpt-5.6-terra), Mage-CUA-Forge replaces per-task agentic construction loops with reusable skill-conditioned workflows and shared application preparers:

1. Browser Transfer & Deterministic Verification: WebVoyager-Static (58.6%) — click to expand

Live web evaluations suffer from content drift, anti-bot blocks, and screenshot-only VLM judges that reward fluent guessing without backend state verification (causing 12.8 to 33.1 pp of score inflation across published web agents). We evaluate browser transfer on WebVoyager-Static — a deterministic offline benchmark reconstructing all 643 tasks across 15 websites on resettable self-hosted replicas, grading both required UI navigation traces and backend SQLite database state:

Even before Stage 5 RL, our generalist desktop-and-browser SFT policy (58.6%) leads among 4B general-purpose models on deterministic offline web evaluation. Whereas Fara1.5-4B is specialized exclusively for browser DOM/screenshot workflows (scoring 13.9% on full-desktop OSWorld-Verified), Mage-VL-Cowork delivers unified competence across both full Linux desktops (66.0%) and multi-step web environments (58.6%) within a single compact 4B model.

🎬 Real-Time Side-by-Side Speedups (4B Hybrid vs. 27B GUI-Only)#

Same task, same desktop, same wall-clock timer against Qwen3.6-27B (GUI-only, thinking mode):

Watch the real-time videos on our Project Page.

🚀 Quick Start#

Important. Custom Architecture Notice: microsoft/Mage-VL-Cowork uses a custom multimodal architecture (model_type: "llava_onevision2", defined in the bundled configuration_llava_onevision2.py and modeling_llava_onevision2.py with a Qwen2_5_VLProcessor image processor). Always pass trust_remote_code=True when loading with transformers or SGLang, and do not assume drop-in weight compatibility with standard Qwen3-VL classes.

1. Installation#

pip install "transformers>=4.57.3" accelerate pillow torch torchvision qwen-vl-utils

2. Serve with SGLang (Docker Runtime Recommended)#

We provide a pre-built, dependency-pinned GPU inference container (docker.io/xiaoyizhang/mage-vl-cowork:0.1.0, featuring SGLang 0.5.2, PyTorch 2.8.0+cu128, Transformers 5.7.0, and FlashInfer 0.3.1 with native llava_onevision2 multimodal support):

export MODEL_DIR=/path/to/Mage-VL-Cowork

docker pull xiaoyizhang/mage-vl-cowork:0.1.0

docker run --rm --name mage-vl-cowork \
  --gpus 'device=0' --shm-size=8g \
  -p 127.0.0.1:30001:30001 \
  --mount "type=bind,src=${MODEL_DIR},dst=/models/Mage-VL-Cowork,readonly" \
  xiaoyizhang/mage-vl-cowork:0.1.0

Alternatively, if launching SGLang directly from a custom environment (bfloat16, 32,768 context window, and SGLANG_MAX_PIXELS=4000000 to preserve full 1920×1080 desktop fidelity):

export SGLANG_MAX_PIXELS=4000000

python -m sglang.launch_server \
  --model-path microsoft/Mage-VL-Cowork \
  --served-model-name Mage-VL-Cowork \
  --host 0.0.0.0 \
  --port 30001 \
  --trust-remote-code \
  --dtype bfloat16 \
  --context-length 32768 \
  --mem-fraction-static 0.82 \
  --tool-call-parser qwen25 \
  --attention-backend flashinfer \
  --mm-attention-backend triton_attn

Tip. Multimodal Request Protocol (/generate): When querying the SGLang server with desktop screenshots, render prompts via AutoProcessor.apply_chat_template (or explicit <|vision_start|><|image_pad|><|vision_end|> vision placeholders) and send base64 image payloads to SGLang's native POST /generate endpoint (image_data). Our bundled MAGEHybridAgent uses this native /generate pipeline by default.

Recommended Sampling Parameters (OSWorld-Verified 66.0% Configuration):

• temperature: 0.7 (or 0.0 for deterministic single-run deployment)
• top_p: 0.8
• max_new_tokens: 4096 (shared budget across <think>, Action:, and <tool_call>)

3. System Prompt & Tool Schema#

Mage-VL-Cowork is trained against a unified two-channel contract (computer_use and bash). Use the exact system prompt below for best performance:

Click to expand the verbatim MAGE_HYBRID_SYSTEM_PROMPT

You are a multi-purpose intelligent assistant. Based on my requests, you can use tools to help me complete various tasks.

# Tools

You have access to the following functions:

<tools>
{"type": "function", "function": {"name": "computer_use", "description": "Use a mouse and keyboard to interact with a computer screen.", "parameters": {"properties": {"action": {"description": "* `click`: Click at (x,y). Use `button` for left/right/middle.\n* `double_click`: Double click at (x,y).\n* `triple_click`: Triple click at (x,y) to select line/paragraph.\n* `move`: Move cursor to (x,y) without clicking.\n* `drag`: Drag from `start` to `end` coordinates.\n* `type`: Type text string.\n* `keypress`: Press key(s). Single key or combo.\n* `scroll`: Scroll at optional (x,y) position. scroll_x/scroll_y in [-999,999], |999|=full screen. Negative scroll_y=down, negative scroll_x=left.\n* `wait`: Wait specified seconds.\n* `screenshot`: Capture the current screen without changing anything.\n* `terminate`: End the task.", "enum": ["click", "double_click", "triple_click", "move", "drag", "type", "keypress", "scroll", "wait", "screenshot", "terminate"], "type": "string"}, "coordinate": {"description": "(x,y) in [0,999].", "type": "array"}, "button": {"description": "Mouse button for click: left/right/middle.", "type": "string", "enum": ["left", "right", "middle"]}, "start": {"description": "(x,y) drag start in [0,999].", "type": "array"}, "end": {"description": "(x,y) drag end in [0,999].", "type": "array"}, "keys": {"description": "Key list for keypress.", "type": "array"}, "text": {"description": "Text for action=type.", "type": "string"}, "scroll_x": {"description": "Horizontal scroll [-999,999]. Negative=left.", "type": "number"}, "scroll_y": {"description": "Vertical scroll [-999,999]. Negative=down.", "type": "number"}, "time": {"description": "Seconds for wait.", "type": "number"}, "status": {"description": "success/failure for terminate.", "type": "string", "enum": ["success", "failure"]}, "msg_to_user": {"description": "Optional message for terminate.", "type": "string"}}, "required": ["action"], "type": "object"}}}
{"type": "function", "function": {"name": "bash", "description": "Run a shell command on the machine and read its output. Use it for file inspection and edits, and for anything the shell does more reliably than the GUI; the result comes back as text, not as a screenshot.", "parameters": {"properties": {"command": {"description": "The shell command to run.", "type": "string"}}, "required": ["command"], "type": "object"}}}
</tools>

For every step, reply in EXACTLY this format with NO suffix:

<think>
your reasoning paragraph describing the current screen state, how it connects to the task goal, and the rationale for the chosen action
</think>

Action: one short imperative sentence describing the next move.

<tool_call>
{"name": "example_function_name", "arguments": {"example_parameter_1": "value_1"}}
</tool_call>

<IMPORTANT>
Reminder:
- Function calls MUST follow the specified format
- Required parameters MUST be specified
- Reasoning goes inside <think>...</think> BEFORE the Action line; nothing after the last tool_call
- You may output at most 15 <tool_call> blocks per response
- Use `type` for text input, `keypress` for shortcuts and special keys
- Every tool_call in one response must use the same function
</IMPORTANT>

# Response format

For every step:
1) <think>...</think> block with your reasoning.
2) Action: a short imperative sentence.
3) One or more <tool_call>...</tool_call> blocks (max 15).

Rules:
- The screen's resolution is 1000x1000. Coordinates are in [0, 999] range.
- Scroll amounts are normalized: |999| = one full screen width/height.
- For scroll, you may include coordinate to scroll at that position.
- For drag, specify start and end coordinates.
- Choose `bash` when the shell is the reliable way to do or check something, and `computer_use` when the work has to happen in the interface on screen.
- A `bash` call is answered with the command's text output; a `computer_use` call is answered with a new screenshot.
- Be brief: one sentence for Action.
- Do not output anything after the last tool call.
- If finishing, use action=terminate.

4. Offline Transformers Inference Example#

import torch
from PIL import Image
from transformers import AutoModelForCausalLM, AutoProcessor

model_path = "microsoft/Mage-VL-Cowork"

processor = AutoProcessor.from_pretrained(model_path, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    model_path,
    trust_remote_code=True,
    torch_dtype=torch.bfloat16,
    device_map="auto",
).eval()

screenshot = Image.open("desktop_screenshot.png").convert("RGB")
task_instruction = (
    "Merge cells A1:D1 in /home/user/Desktop/budget.xlsx, "
    "set the title to 'Q4 Summary', and save the file."
)

messages = [
    {"role": "system", "content": MAGE_HYBRID_SYSTEM_PROMPT},
    {
        "role": "user",
        "content": [
            {"type": "image"},
            {"type": "text", "text": f"Task: {task_instruction}"},
        ],
    },
]

prompt = processor.apply_chat_template(
    messages, tokenize=False, add_generation_prompt=True
)
inputs = processor(text=[prompt], images=[screenshot], return_tensors="pt").to(model.device)
if "pixel_values" in inputs:
    inputs["pixel_values"] = inputs["pixel_values"].to(model.dtype)

with torch.inference_mode():
    output_ids = model.generate(
        **inputs,
        max_new_tokens=4096,
        temperature=0.7,
        top_p=0.8,
        do_sample=True,
    )

response = processor.tokenizer.decode(
    output_ids[0, inputs["input_ids"].shape[1]:], skip_special_tokens=True
)
print(response)

Example model output:

<think>
The user wants to merge cells A1:D1 in `/home/user/Desktop/budget.xlsx` and set the banner text to 'Q4 Summary'. Currently the desktop is open, and LibreOffice Calc is not running. Since this is a structured spreadsheet manipulation on an existing `.xlsx` file, editing it directly with `openpyxl` in `bash` is much faster and more reliable than opening Calc and clicking through cell menus.
</think>

Action: Use Python with openpyxl in bash to merge A1:D1, set the title to 'Q4 Summary', and save `/home/user/Desktop/budget.xlsx`.

<tool_call>
{"name": "bash", "arguments": {"command": "python3 -c \"\nimport openpyxl\nwb = openpyxl.load_workbook('/home/user/Desktop/budget.xlsx')\nws = wb.active\nws.merge_cells('A1:D1')\nws['A1'] = 'Q4 Summary'\nwb.save('/home/user/Desktop/budget.xlsx')\nprint('Saved successfully')\n\""}}
</tool_call>

5. Reproducing OSWorld-Verified Evaluation (osworld_eval/)#

This repository bundles the minimal reference package in osworld_eval/ containing the SGLang Docker launcher and the core Mage-VL-Cowork agent definitions (MAGEHybridAgent and MAGEGUIAgent):

osworld_eval/
├── README.md
├── mm_agents/
│   ├── mage_gui_agent.py       # Base GUI computer_use agent & normalized [0, 999] action parser
│   └── mage_hybrid_agent.py    # Hybrid GUI (computer_use) + Bash agent & SGLang /generate client
└── scripts/
    └── serve_sglang_docker.sh  # SGLang Docker inference server launcher

Step 1: Launch the SGLang Inference Runtime#

Start the GPU inference server using the pre-built container (docker.io/xiaoyizhang/mage-vl-cowork:0.1.0):

export MODEL_DIR=/path/to/Mage-VL-Cowork
bash "${MODEL_DIR}/osworld_eval/scripts/serve_sglang_docker.sh"

Step 2: Run the OSWorld-Verified Evaluation Harness#

For the complete end-to-end OSWorld-Verified evaluation harness (361 tasks in test_nogdrive.json, 50-step budget), follow the instructions in our official GitHub repository: https://github.com/microsoft/Mage/tree/main/mage_vl_cowork, or integrate osworld_eval/mm_agents/ directly into your OSWorld checkout.

Standard Evaluation Configuration (66.0% Pipeline)#

See osworld_eval/README.md and the GitHub repository (mage_vl_cowork) for complete documentation.

🛡️ Responsible AI & Safe Deployment#

Mage-VL-Cowork is an autonomous hybrid agent capable of executing arbitrary shell commands (bash) and GUI interactions (computer_use). It is designed and evaluated for isolated, disposable sandbox environments (such as Docker containers or snapshot-reverted virtual machines). When deploying Mage-VL-Cowork, follow these operational best practices:

• Mandatory Sandbox Isolation: Always run the agent inside an isolated container or VM with no access to host filesystems, local networks, or personal environment variables.
• Scoped Authorization & Human-in-the-Loop: Require explicit human confirmation before permitting irreversible actions (e.g., external network submissions, account authentication, financial transactions, or privilege modifications).
• Zero Credential Exposure: Do not mount personal SSH keys, cloud tokens, browser cookies, or password stores inside the agent's workspace unless explicitly required and scoped to the sandbox.
• Network Allow-Listing: Restrict outbound network access when evaluating untrusted web content to mitigate indirect prompt injection embedded in external web pages or documents.

🏁 Summary#

Mage-VL-Cowork demonstrates that an efficient 4B small vision-language model can master hybrid computer use when paired with upstream teacher-harness evolution, workflow-first environment synthesis (Mage-CUA-Forge), and a five-stage specialize-then-unify curriculum. Achieving the best performance at the 4B scale (66.0% on OSWorld-Verified) while using 13.0% fewer model calls and 25.4% fewer output tokens after online RL, Mage-VL-Cowork pushes the frontier of on-device intelligence for autonomous desktop agents.

📝 Citation#

@article{zhang2026magecowork,
  title={Mage-VL-Cowork: A Full Training Recipe to Learn Hybrid Computer Use from Evolved Harnesses},
  author={Zhang, Xiaoyi and Wang, Haoqing and Zhang, Xinjie and Ming, Xiang and Wang, Xing and Yang, Senqiao and Zhang, Kaichen and Jia, Zhaoyang and Tan, Andong and Wang, Xinjiang and Guo, Jinghao and Shen, Yifei and Li, Xiao and Zhang, Peng and Guo, Xun and Li, Jiahao and Zheng, Shicheng and Wang, Jinglu and Guo, Zongyu and Xie, Wenxuan and Li, Bin and Lu, Yan},
  journal={arXiv preprint},
  year={2026}
}

📄 License#

Mage-VL-Cowork is released under the Apache-2.0 License.