LFM3-Omni-2B

Image: Liquid AI

API Try Docs Blog

LFM3-Omni-2B#

LFM3-Omni-2B is Liquid AI's first omni-modal model in the LFM3 family: a single end-to-end model that understands text, images, video and audio, and answers in text and speech. It is built for on-device use: one model replaces a separate ASR model, vision-language model, LLM and TTS model.

• Any input, text or speech output: ask about a photo by voice, transcribe or translate speech, chat out loud, or ask questions about a video and its soundtrack together.
• Watches and listens together: for videos with sound, the model sees and hears them at the same time, so it can answer questions that need both.
• Small enough for devices: at 2.2B parameters, it is designed to run efficiently on phones, laptops and other edge hardware.

Find more information about the LFM3 family in our release blog post.

Model details#

The LFM3 family includes base, text, vision, audio, omni, and decision models.

Image: LFM3-Omni architecture: a spoken question passes through the LFM3 audio encoder and audio connector, an image through the LFM3 vision encoder and vision connector, and a text prompt through the tokenizer and embedding, all into the LFM3-1.7B Liquid Foundation Model. Its output stream of text and audio tokens drives the LM head for text and the depthformer and Liquid Audio Codec for 24 kHz speech.

| Property | Specification

| Checkpoint | Post-trained omni-modal model

| Parameters | 2.17B total

| Architecture | LFM3 backbone, LFM3 vision encoder, LFM3 causal audio encoder, LiquidCodec audio tokenizer.

| Inputs | Text, images, video with or without its soundtrack, and 16 kHz audio. Images and video are processed at native resolution; each 2×2 block of patches becomes one language token. Video is sampled at 2 fps, with 4 to 384 frames by default.

| Outputs | Text, 24 kHz speech, or interleaved text and speech

| Context length | 32K tokens (input + output)

| Vocabulary | 128,000 text tokens

| Text languages | English, Arabic, Chinese, French, German, Italian, Japanese, Korean, Portuguese, Spanish, Vietnamese, Thai, Indonesian, Hindi, Russian, Polish

| Speech languages and locales | Arabic, Danish, German, Greek, British English, American English, Finnish, French, Hebrew, Italian, Japanese, Korean, Dutch, Polish, Portuguese, Spanish, Swedish, Turkish, Chinese

| License | LFM Open License v1.0

This is a non-reasoning model. enable_thinking=True is not supported.

Inference#

Choose a checkpoint variant for your inference framework and hardware.

| Checkpoint | Framework | Description

| BF16 | SGLang, Transformers | BF16 weights for GPU inference. SGLang serves text output.

| GGUF | llama.cpp | Local inference on CPUs and GPUs, including a realtime speech server.

Sampling parameters#

• Text output: greedy decoding works well.
• Speech or interleaved output: use audio_temperature=0.4, audio_repetition_penalty=1.2

Generation modes (optional)#

To force a generation mode, pass the optional generation_mode argument to model.generate.

| Mode | generation_mode (optional) | Opening token

| Text | "text" | none

| Speech | "audio" | <|audio_start|>

| Text and speech, interleaved | "interleaved" | <|mixed_start|>

By default, the model selects the output modality based on the system prompt and input modality, typically matching the input modality. To encourage a specific output modality, append the corresponding instructions to the end of your system prompt.

| Mode | System Prompt

| Text | Respond in text.

| Text and speech, interleaved | Respond in interleaved text and audio.

Transformers quick start#

Install the dependencies, then load the model and processor with the checkpoint's custom code. The codec implementation is included in this repository, and the processor loads its weights from audio_tokenizer/.

pip install "transformers>=5.20.2" torch torchvision torchcodec accelerate pillow librosa soundfile

import torch
import soundfile as sf

from transformers import AutoModelForMultimodalLM, AutoProcessor

model_id = "LiquidAI/LFM3-Omni-2B"
device = "cuda" if torch.cuda.is_available() else "cpu"
dtype = torch.bfloat16 if device == "cuda" else torch.float32

model = AutoModelForMultimodalLM.from_pretrained(
    model_id, dtype=dtype, device_map=device, trust_remote_code=True
).eval()
processor = AutoProcessor.from_pretrained(model_id, trust_remote_code=True)
codec = processor.audio_tokenizer.to(device=device).eval()

Chat template

LFM3-Omni uses a ChatML-like format, which processor.apply_chat_template() renders for you, generated from the checkpoint's chat template:

<|startoftext|><|im_start|>system
Respond with text.<|im_end|>
<|im_start|>user
Name a famous bridge in San Francisco.<|im_end|>
<|im_start|>assistant
<think></think>The Golden Gate Bridge.<|im_end|>
<|im_start|>user
When did it open?<|im_end|>
<|im_start|>assistant
<think></think>

The system turn is optional, and every assistant turn opens with an empty <think></think>. Each image, video or audio clip is wrapped in <|vision_start|> / <|vision_end|>, which the template adds. Here is an image and a spoken question:

<|startoftext|><|im_start|>user
<|vision_start|><|image|><|vision_end|><|vision_start|><|audio|><|vision_end|>Answer in one sentence.<|im_end|>
<|im_start|>assistant
<think></think>

Usage examples#

The examples below reuse the model, codec, and processor loaded above. Replace audio.wav, question.wav, recording.wav, photo.jpg, or clip.mp4 with your own files. Audio files are resampled to 16 kHz mono on load.

Speech recognition (ASR)

To use the model for transcription, use the system prompt Perform ASR in <language>., where <language> is the English name of the language.

Below, we transcribe a .wav file containing English speech.

conversation = [
    {"role": "system", "content": "Perform ASR in English."},
    {"role": "user", "content": [{"type": "audio", "audio": "audio.wav"}]},
]

inputs = processor.apply_chat_template(
    conversation,
    add_generation_prompt=True,
    tokenize=True,
    return_dict=True,
    return_tensors="pt",
).to(device)
with torch.no_grad():
    outputs = model.generate(**inputs, max_new_tokens=1024, generation_mode="text")
text = processor.decode(
    outputs.sequences[0, inputs["input_ids"].shape[1]:outputs.sequence_lengths[0]],
    skip_special_tokens=True,
)
print(text)

Speech synthesis (TTS)

For speech synthesis, use the system prompt Perform TTS in <language> using a <gender> voice.. Use one of the speech languages listed in Model details and a female or male voice. For English, say American English or British English. Pass generation_mode="audio" so the answer opens with <|audio_start|>.

conversation = [
    {"role": "system", "content": "Perform TTS in American English using a female voice."},
    {"role": "user", "content": "The quick brown fox jumps over the lazy dog"},
]
inputs = processor.apply_chat_template(
    conversation,
    add_generation_prompt=True,
    tokenize=True,
    return_dict=True,
    return_tensors="pt",
).to(device)
with torch.no_grad():
    outputs = model.generate(
        **inputs,
        max_new_tokens=256,
        generation_mode="audio",
        audio_temperature=0.4,
        audio_repetition_penalty=1.2,
    )
    waveform = codec.decode(
        outputs.audio_codes[:1, :, :outputs.audio_lengths[0]]
    ).audio_values[0]  # 24 kHz

sf.write("tts.wav", waveform.squeeze(0).float().cpu().numpy(), 24_000)

Voice chat: speech and text responses

In this mode, the model will respond to the user in both speech and text. The recommended system prompt is Respond with interleaved text and audio.. User input can be either text or audio, and we pass generation_mode="interleaved" so the answer opens with <|mixed_start|>.

The assistant responds in a conversational style with interleaved text and speech. This example saves the complete response.

conversation = [
    {"role": "system", "content": "Respond with interleaved text and audio."},
    {"role": "user", "content": [{"type": "audio", "audio": "question.wav"}]},
]
inputs = processor.apply_chat_template(
    conversation,
    add_generation_prompt=True,
    tokenize=True,
    return_dict=True,
    return_tensors="pt",
).to(device)
with torch.no_grad():
    outputs = model.generate(
        **inputs,
        max_new_tokens=1024,
        generation_mode="interleaved",
        audio_temperature=0.4,
        audio_repetition_penalty=1.2,
    )

    text = processor.decode(
        outputs.sequences[0, inputs["input_ids"].shape[1]:outputs.sequence_lengths[0]],
        skip_special_tokens=True,
    )
    waveform = codec.decode(
        outputs.audio_codes[:1, :, :outputs.audio_lengths[0]]
    ).audio_values[0]  # 24 kHz
print(text)
sf.write("reply.wav", waveform.squeeze(0).float().cpu().numpy(), 24_000)

Speech-to-text chat

For a text-only answer, no specific system prompt is needed. Respond with text. makes it explicit. Pass generation_mode="text". LFM3-Omni-2B is a non-reasoning model: every answer opens with an empty <think></think>.

conversation = [
    {"role": "user", "content": [{"type": "audio", "audio": "question.wav"}]},
]
inputs = processor.apply_chat_template(
    conversation,
    add_generation_prompt=True,
    tokenize=True,
    return_dict=True,
    return_tensors="pt",
).to(device)
with torch.no_grad():
    outputs = model.generate(**inputs, max_new_tokens=1024, generation_mode="text")
    text = processor.decode(
        outputs.sequences[0, inputs["input_ids"].shape[1]:outputs.sequence_lengths[0]],
        skip_special_tokens=True,
    )
print(text)

Audio understanding

Include the recording and a question in the user turn. Questions can address speech, music, background sounds, or events. No specific system prompt is required. To ground answers in the recording, use Answer questions about the provided audio clip. Base every answer only on what can be heard..

conversation = [
    {
        "role": "system",
        "content": "Answer questions about the provided audio clip. Base every answer only on what can be heard.",
    },
    {
        "role": "user",
        "content": [
            {"type": "audio", "audio": "recording.wav"},
            {"type": "text", "text": "What is happening in this recording?"},
        ],
    },
]
inputs = processor.apply_chat_template(
    conversation,
    add_generation_prompt=True,
    tokenize=True,
    return_dict=True,
    return_tensors="pt",
).to(device)

with torch.no_grad():
    outputs = model.generate(**inputs, max_new_tokens=1024, generation_mode="text")

print(processor.decode(
    outputs.sequences[0, inputs["input_ids"].shape[1]:outputs.sequence_lengths[0]],
    skip_special_tokens=True,
))

For multiple-choice questions, include the choices in the text part and end it with Please choose the answer from the options.

Translation

Use a system prompt that specifies the source language, target language, and output format:

• Speech to text: Please translate the following spoken {source} into {target} text.

Replace {source} and {target} with the English names of the source and target languages. For text input, set the user message's content to the text to translate. Translation to or from English generally works best.

Image understanding

Include the image and a question in the user turn. To ground answers in the image, use the system prompt Answer questions about the provided image. Base every answer only on what can be seen.. For captions, use Describe the provided image.. Images are processed at their native resolution, up to 2,048 tokens per image.

conversation = [
    {
        "role": "system",
        "content": "Answer questions about the provided image. Base every answer only on what can be seen.",
    },
    {
        "role": "user",
        "content": [
            {"type": "image", "image": "photo.jpg"},
            {"type": "text", "text": "What is written on the sign?"},
        ],
    },
]
inputs = processor.apply_chat_template(
    conversation,
    add_generation_prompt=True,
    tokenize=True,
    return_dict=True,
    return_tensors="pt",
).to(device)

with torch.no_grad():
    outputs = model.generate(**inputs, max_new_tokens=1024, generation_mode="text")

print(processor.decode(
    outputs.sequences[0, inputs["input_ids"].shape[1]:outputs.sequence_lengths[0]],
    skip_special_tokens=True,
))

Spoken questions about an image

The question can be asked by voice, with the recording in place of the text. Add Respond with text. to the system prompt for a text answer, or follow the voice chat example with Respond with interleaved text and audio. for a spoken one.

conversation = [
    {
        "role": "system",
        "content": "Answer questions about the provided image. Base every answer only on what can be seen. Respond with text.",
    },
    {
        "role": "user",
        "content": [
            {"type": "image", "image": "photo.jpg"},
            {"type": "audio", "audio": "question.wav"},
        ],
    },
]
inputs = processor.apply_chat_template(
    conversation,
    add_generation_prompt=True,
    tokenize=True,
    return_dict=True,
    return_tensors="pt",
).to(device)

with torch.no_grad():
    outputs = model.generate(**inputs, max_new_tokens=1024, generation_mode="text")

print(processor.decode(
    outputs.sequences[0, inputs["input_ids"].shape[1]:outputs.sequence_lengths[0]],
    skip_special_tokens=True,
))

Video understanding

Include the video and a question in the user turn. No specific system prompt is required. Video is sampled at 2 frames per second (4 to 384 frames), and each pair of frames is prefixed with its timestamp.

conversation = [
    {
        "role": "user",
        "content": [
            {"type": "video", "video": "clip.mp4"},
            {"type": "text", "text": "What happens in this video?"},
        ],
    },
]
inputs = processor.apply_chat_template(
    conversation,
    add_generation_prompt=True,
    tokenize=True,
    return_dict=True,
    return_tensors="pt",
).to(device)

with torch.no_grad():
    outputs = model.generate(**inputs, max_new_tokens=1024, generation_mode="text")

print(processor.decode(
    outputs.sequences[0, inputs["input_ids"].shape[1]:outputs.sequence_lengths[0]],
    skip_special_tokens=True,
))

Video with its soundtrack

To let the model see and hear a video, use the system prompt Answer questions about the video using what you see and hear. and pass two flags:

• load_audio_from_video=True reads the audio track out of the video file.
• use_audio_in_video=True interleaves that audio with the frames, so each pair of frames is followed by the sound that goes with it.

Without them, the model gets the frames only.

conversation = [
    {"role": "system", "content": "Answer questions about the video using what you see and hear."},
    {
        "role": "user",
        "content": [
            {"type": "video", "video": "clip.mp4"},
            {"type": "text", "text": "What happens here, and what can you hear?"},
        ],
    },
]
inputs = processor.apply_chat_template(
    conversation,
    add_generation_prompt=True,
    tokenize=True,
    return_dict=True,
    return_tensors="pt",
    load_audio_from_video=True,
    processor_kwargs={"use_audio_in_video": True},
).to(device)

with torch.no_grad():
    outputs = model.generate(**inputs, max_new_tokens=1024, generation_mode="text")

print(processor.decode(
    outputs.sequences[0, inputs["input_ids"].shape[1]:outputs.sequence_lengths[0]],
    skip_special_tokens=True,
))

Video frame rate and frame limit

By default, LFM3-Omni-2B samples video at 2 fps, with at least 4 and at most 384 frames per video. Each frame adds vision tokens, so for long videos a lower frame rate or frame limit uses less context and memory. These settings do not change the soundtrack: the model always hears all of it.

• Transformers: set the frame rate per call with processor.apply_chat_template(..., processor_kwargs={"fps": 1}). With the soundtrack, use processor_kwargs={"use_audio_in_video": True, "fps": 1}. Set the frame limit when you load the processor: AutoProcessor.from_pretrained(model_id, trust_remote_code=True, max_frames=128).
• SGLang: change fps, min_frames, and max_frames under video_processor in processor_config.json in a local copy of the checkpoint.
• llama.cpp: always uses 2 fps and 4 to 384 frames for this model. --video-fps has no effect.

Tool calling from spoken requests

Pass function schemas through tools=... to apply_chat_template(). The chat template places them in the system prompt between <tools> and </tools>. Requests may be spoken or written. A tool call looks like this:

<tool_call>
<function=set_volume>
<parameter=volume>
30
</parameter>
</function>
</tool_call>

Values other than strings are written as JSON. Your application parses and runs the calls, then appends the assistant's tool_calls (with arguments as a dict) and a tool message with the results to the conversation. Generate again with the same tools: the model either makes more calls or answers.

import json
import re

tools = [{
    "type": "function",
    "function": {
        "name": "set_volume",
        "description": "Set the device speaker volume.",
        "parameters": {
            "type": "object",
            "properties": {
                "volume": {"type": "integer", "description": "Volume from 0 to 100."}
            },
            "required": ["volume"],
        },
    },
}]
functions = {"set_volume": lambda volume: {"volume": volume}}  # your implementations

CALL_RE = re.compile(r"<tool_call>\s*<function=([^>\n]+)>(.*?)</function>\s*</tool_call>", re.DOTALL)
PARAM_RE = re.compile(r"<parameter=([^>\n]+)>\n?(.*?)\n?</parameter>", re.DOTALL)


def parse_tool_calls(text):
    calls = []
    for name, body in CALL_RE.findall(text):
        arguments = {}
        for key, value in PARAM_RE.findall(body):
            try:
                arguments[key] = json.loads(value)
            except json.JSONDecodeError:
                arguments[key] = value  # plain string
        calls.append({"type": "function", "function": {"name": name, "arguments": arguments}})
    return calls


def respond(conversation):
    inputs = processor.apply_chat_template(
        conversation,
        tools=tools,
        add_generation_prompt=True,
        tokenize=True,
        return_dict=True,
        return_tensors="pt",
    ).to(device)
    with torch.no_grad():
        outputs = model.generate(**inputs, max_new_tokens=1024, generation_mode="text")
    return processor.decode(
        outputs.sequences[0, inputs["input_ids"].shape[1]:outputs.sequence_lengths[0]],
        skip_special_tokens=True,
    )


conversation = [
    {"role": "user", "content": [{"type": "audio", "audio": "question.wav"}]},
]
text = respond(conversation)
while calls := parse_tool_calls(text):
    results = [functions[c["function"]["name"]](**c["function"]["arguments"]) for c in calls]
    conversation += [
        {"role": "assistant", "tool_calls": calls},
        {"role": "tool", "content": json.dumps(results)},
    ]
    text = respond(conversation)

print(text)

See the Tool Use documentation for details.

Fine-tuning#

LFM3-Omni-2B is designed for easy customization for a given domain, task, or application with Hugging Face Transformers.

| Training | Use case | Notebook

| SFT | Fine-tune on your own omni data (text, audio, images) with Transformers | Open in Colab

| SFT | Visual question answering | Open in Colab

Performance#

Image: LFM3-Omni radar chart: one model scored across text (IFEval, GSM8K, MMLU-Pro, BFCL v4, RULER), vision (general, perception, documents, math and reasoning, grounding, embodied, GUI, video, multilingual), speech (English and multilingual ASR, TTS quality, VoiceBench, MMAR, voice tool calling) and omni tasks (image and audio, video and audio, spoken VQA, spoken embodied).

Omni#

| Benchmark | LFM3-Omni 2B | Qwen2.5-Omni 3B | Qwen2.5-Omni 7B | Gemma 4 E2B | Gemma 4 E4B | Phi-4 multimodal

| Total parameters | 2.2B | 6B | 10.7B | 5B | 8B | 5.6B

| Video + audio understanding | |

| Daily-Omni | 70.09 | 58.81 | 61.32 | 50.63 | 54.14 | 48.87

| WorldSense | 47.10 | 44.99 | 45.37 | 38.11 | 43.51 | 42.37

| AV-SpeakerBench | 46.79 | 40.44 | 43.24 | 32.32 | 36.36 | 37.95

| Image + audio understanding | |

| OmniBench (overall) | 46.32 | 47.11 | 49.65 | 16.81 | 20.67 | 38.97

| OmniBench (speech) | 48.25 | 45.65 | 47.99 | 16.08 | 21.27 | 38.91

| Spoken visual QA | |

| RealWorldQA | 64.68 | 61.05 | 69.93 | 39.35 | 49.02 | 64.31

| MMStar | 51.79 | 50.80 | 57.27 | 38.87 | 44.60 | 54.80

| POPE | 83.78 | 85.34 | 85.37 | 74.20 | 77.82 | 83.00

| ERQA | 39.43 | 37.34 | 39.16 | 20.63 | 32.12 | 30.81

| EmbSpatial | 67.59 | 53.40 | 59.51 | 44.08 | 54.01 | 42.60

Vision#

| Benchmark | LFM3-Omni 2B | Qwen2.5-Omni 3B | Gemma 4 E2B | Gemma 4 E4B | Phi-4 multimodal | Cosmos3 Edge

| Total parameters | 2.2B | 6B | 5B | 8B | 5.6B | 4B

| General | |

| MMStar | 55.67 | 55.67 | 46.47 | 52.00 | 59.33 | 56.07

| MMMU (val) | 42.44 | 48.00 | 37.81 | 43.14 | 54.76 | 37.14

| MMMU-Pro | 27.28 | 24.80 | 24.34 | 28.67 | 27.63 | 13.76

| MMBench | 73.68 | 79.30 | 67.61 | 73.54 | 82.73 | 76.63

| SEEDBench (image) | 75.07 | 75.27 | 72.53 | 74.78 | 72.24 | 74.73

| RealWorldQA | 66.54 | 63.14 | 48.63 | 53.07 | 64.84 | 72.81

| MathVista (mini) | 62.50 | 58.80 | 51.00 | 54.30 | 65.40 | 53.90

| POPE | 89.34 | 84.62 | 78.78 | 81.24 | 83.34 | 83.52

| Documents and OCR | |

| DocVQA | 92.47 | 92.71 | 73.49 | 77.91 | 92.60 | 84.43

| InfoVQA | 64.08 | 74.55 | 38.72 | 48.30 | 65.98 | 60.94

| OCRBench | 806 | 863 | 731 | 761 | 795 | 751

| Grounding | |

| RefCOCO | 73.55 | 78.76 | 63.18 | 63.36 | —* | 74.53

| Video | |

| MVBench | 58.03 | 66.43

The publisher's README is longer; it is cut here. Read the rest at the source.