Orca is an initial instantiation of a general world foundation model. It learns a unified world latent space from multimodal world signals and exposes the learned latent through multimodal readout interfaces.
Rather than optimizing isolated next-token, next-frame, or next-action prediction objectives, Orca is centered on Next-State-Prediction: a unified state-transition modeling route toward understanding, predicting, and acting upon the world. In this version, Orca focuses on two fundamental input signals: visual signals for dense observations of world evolution, and language signals for event descriptions, task intentions, causal explanations, and semantic constraints.
Unconscious learning: dense natural transitions from continuous videos.
Conscious learning: sparse meaningful transitions under language-described events and VQA supervision.
Frozen-backbone readouts: lightweight decoders for text, images, and actions.
Scaling analysis: stronger world modeling, stronger downstream readouts.
Orca follows an Encoder-Decoder architecture. Given multimodal world signals, the Encoder learns a world latent through unconscious and conscious learning. After pre-training, the Encoder is frozen, and only lightweight modality-specific decoders are trained to read out the latent into downstream modalities.
Figure 2. Lightweight readouts adapt frozen world latents to language, vision, and action.
Orca models world-state transitions under both implicit dynamics and explicit conditions. Implicit dynamics capture latent or unobserved factors such as physical laws, object properties, scene dynamics, and environmental forces, while explicit conditions describe observed signals such as human instructions, event descriptions, task intentions, or causal premises.
For pre-training, Orca constructs a large-scale world-learning inventory from visual signals and language signals. The data mixture includes video data for observation-only state transitions, event data for event-conditioned state transitions, and VQA data for response generation.
125K hours of video data covering egocentric interaction, exocentric manipulation, robot execution, and natural dynamics.
160M event annotations with fine- and coarse-grained captions for event-level transition learning.
General VQA data for aligning world latents with language understanding and response generation.
Figure 4. Downstream readout performance improves as Orca pre-training scales.
Experiments indicate that stronger world latents from pre-training lead to stronger downstream readouts. As pre-training scales up, Orca improves across text, image, and action readouts while keeping the backbone frozen during readout post-training.
If you find Orca useful for your research, please consider citing our technical report.
@article{orca2026,
title={Orca: The World is in Your Mind},
author={Yihao Wang and Yuheng Ji and Mingyu Cao and Yanqing Shen and Runze Xiao and Huaihai Lyu and Senwei Xie and Euan Liu and Klara Tian and Tianfeng Long and Yichi Zhang and Zhengliang Cai and Ruike Chen and Jifan Zhao and Ruochuan Shi and Zihan Tang and Jing Lyu and Wenxing Tan and Ningbo Zhang and Yangtao Hu and Yuming Gao and Xiansheng Chen and Junkai Zhao and Congsheng Xu and Boan Zhu and Ziqi Wang and Yupu Feng and Qiongqiong Zhang and Yingli Zhao and Yulong Ao and Shaoxuan Xie and You Liu and Guocai Yao and Leiduo Zhang and Xiaodan Liu and Yunyan Zhang and Yance Jiao and Xinyan Yang and Jiaxing Wei and Xu Liu and Tengfei Pan and Shaokai Nie and Chunlei Men and Sen Cui and Xiaojie Jin and Hongyang Li and Jianlan Luo and Yao Mu and Yunchao Wei and Jun Yan and Hang Zhao and Xiaolong Zheng and Jiaming Li and Yonghua Lin and Tiejun Huang and Zhongyuan Wang and Pengwei Wang},
journal={arXiv preprint arXiv:2606.30534},
year={2026}
}
Files
11 files · 11.0 GBWeights 11.0 GB (safetensors). Sizes and sha256 from the signed manifest.
BAAI <huggingface.co/BAAI>Newsgroups: alt.freeweights.textSubject: 
