Bernini is a unified framework for video generation and editing that combines an MLLM-based semantic planner with a DiT-based renderer.
On video editing, Bernini reaches the first tier among leading closed-source commercial models. The leaderboard below comes from our self-built arena platform, where human annotators blindly vote on paired edits and the votes are aggregated into a Bradley-Terry score and a pairwise win-rate matrix.
CUDA GPU — a Hopper GPU (H100/H800/H200) is recommended so FlashAttention-3 can be used; other CUDA GPUs fall back to FlashAttention-2 or PyTorch SDPA.
CUDA toolkit 12.4 (matches the pinned torch==2.5.1+cu124; 12.3+ is the minimum if you build FlashAttention-3).
Pinned in requirements.txt: torch==2.5.1+cu124, diffusers==0.35.2, accelerate==0.34.2, transformers==4.57.3.
Reference environment (Bernini-R is developed and tested on this setup):
Multi-GPU sequence parallel needs Open-VeOmni (Apache-2.0, Python 3.11). Use --no-deps so VeOmni does not pull in a different torch build and override the pinned torch==2.5.1+cu124: pip install --no-deps git+https://github.com/ByteDance-Seed/VeOmni.git@v0.1.10. Single-GPU inference does not need it.
Faster attention (auto-detected if installed; otherwise PyTorch SDPA is used):
FlashAttention-2 — general CUDA GPUs (incl. A100/A800): pip install flash-attn==2.8.3.
FlashAttention-3 — Hopper only (H100/H800/H200, CUDA ≥ 12.3, PyTorch ≥ 2.4). flash_attn_interface is not on PyPI; build it from the flash-attention repo's hopper/ directory at tag v2.8.3: bash git clone https://github.com/Dao-AILab/flash-attention.git cd flash-attention && git checkout v2.8.3 cd hopper && MAX_JOBS=$(nproc) python3 setup.py install --user
Bernini-R provides two ways to obtain the renderer weights. The diffusers format is recommended — it is a self-contained diffusers-format directory whose transformer / transformer_2 already hold the Bernini-R weights, so you point --config at it and the weights load directly, with no--high_noise_ckpt / --low_noise_ckpt needed.
A single ready-to-use diffusers-format model from ByteDance/Bernini-R-Diffusers. It bundles the Wan2.2 base components (VAE, UMT5 text encoder, tokenizer) together with the Bernini-R transformer weights, so nothing else is downloaded at runtime.
The original layout, where Bernini-R uses two sets of weights loaded separately:
Wan2.2 base — Wan-AI/Wan2.2-T2V-A14B-Diffusers on Hugging Face. Supplies the VAE, UMT5 text encoder, tokenizer, and the transformer architecture/base weights. It is downloaded automatically on first run (configured by wan22_base in configs/bernini_renderer_wan22/config.json).
Bernini-R checkpoint — the trained high-noise / low-noise transformer weights (safetensors) from ByteDance/Bernini-R, passed with --high_noise_ckpt / --low_noise_ckpt. Both a local directory and a Hugging Face repo id are accepted.
A run is described by a case file — a small JSON under assets/testcases/ that bundles one task's routing and inputs (task_type, guidance_mode, prompt, source media, output). This keeps long prompts out of the command line. Each task has a directory under assets/testcases/ holding one or more case files; see assets/testcases/ for the format and the bundled t2i / i2i / t2v / v2v / rv2v /r2v examples.
--use_pe enhances the prompt through an OpenAI-compatible endpoint and is recommended for best generation quality. The openai SDK is installed by requirements.txt; configure the endpoint with environment variables:
export BERNINI_PE_API_KEY=... # or OPENAI_API_KEY
export BERNINI_PE_BASE_URL=... # or OPENAI_BASE_URL
export BERNINI_PE_MODEL=... # vision-capable chat model
Unless an example specifies otherwise, inference outputs 480p / 16fps (the defaults — --max_image_size 848, --fps 16).
Each example runs a bundled case in assets/testcases/ — replace <hi> / <lo> with your high-/low-noise checkpoint paths. The image tasks (t2i, i2i) are shown on a single GPU; the video tasks on 8 GPUs via torchrun, where --ulysses N gives N-way Ulysses sequence parallel per sample and the remaining world_size / N ranks run data parallel over the task list. The two scripts take the same inputs, so any example can be run either way.
Inputs can also be passed directly as flags instead of --case (--prompt, --task_type, --guidance_mode, --video, --image, --images, --output); generation parameters (--seed, --num_frames, ...) are always command-line flags.
Text-to-image (t2i) — single GPU; generates one frame, so pass --num_frames 1
gradio_demo.py exposes the same pipeline through a Gradio UI: the task-type dropdown auto-fills guidance_mode (still user-editable), uploaded media is routed to the matching slot, and the result is rendered inline.
Add --use_pe (and export OPENAI_API_KEY=... / BERNINI_PE_API_KEY=...) to enable GPT prompt enhancement; the in-UI checkbox is a per-request switch on top of this flag.
@article{bernini,
title = {Bernini: Latent Semantic Planning for Video Diffusion},
author = {Chenchen Liu and Junyi Chen and Lei Li and Lu Chi and Mingzhen Sun and Zhuoying Li and Yi Fu and Ruoyu Guo and Yiheng Wu and Ge Bai and Zehuan Yuan},
journal = {arXiv preprint arXiv:2605.22344},
year = {2026}
}
