MiniCPM-OCR-1.0-GPTQ

MiniCPM logo

Usage Guide

| 🚀 Free API

Try it online for free, then call the API from your own code.

Playground (CN / Global)

API docs (CN / Global)

| 🤖 Agent Skill

One prompt: the agent installs it and parses your first document.

SKILL.md

| 💻 Self-hosting

vLLM / SGLang / llama.cpp / Ollama, plus an official Docker image. Data stays on your side.

Deployment guide

| 📱 Mobile App

On-device and offline. Document parsing is iOS-only for now.

iOS Demo

Demo (CN / Global)  ·  GitHub  ·  SDK usage  ·  DSpark  ·  Discord  |  中文

This repository hosts the GPTQ W4A16 quantized version of MiniCPM-OCR 1.0. For the original BF16 weights and the base model card, see openbmb/MiniCPM-OCR-1.0. Everything below describes the model itself; see Quantization for what changed.

MiniCPM-OCR 1.0 (1.3B) is the first OCR-specific model in the MiniCPM-V/o family. It focuses on text-centric document understanding — text, tables, formulas, charts, seals, and structured information extraction — and balances quality against speed on real-world captures.

Highlights#

• 🏆 SOTA document parsing: 93.68 on Real5-OmniDocBench, a benchmark for real-world document parsing, with leading results on OmniDocBench v1.6, Wild-OmniDocBench, and olmOCR-Bench. Handles Chinese and English documents and PDFs, including scans, curved pages, photographs, skewed pages, and recaptured documents.
• ⚡ Efficient inference. With DSpark decoding acceleration (BF16, GGUF) end-to-end processing efficiency improves by 55%. Throughput is about 1.49× PaddleOCR-VL-1.6 and 3.62× MinerU2.5 Pro.
• 🔍 General recognition and information extraction: 54.7 on OCRBench v2, plus key information extraction (KIE) for downstream automation.
• 🛠️ Easy to deploy and develop: vLLM, SGLang, llama.cpp, and Ollama from day one, with quantized variants and multi-chip builds. Supports further fine-tuning, and ships a pip SDK, a full API, and agent workflows. Online demos: China · Global.

SOTA quality at higher throughput#

Among the open-source OCR models compared, MiniCPM-OCR 1.0 reaches 1.2 pages/s and the highest equal-weight average score across OmniDocBench v1.6, Real5-OmniDocBench, Wild-OmniDocBench, and olmOCR-Bench.

Image: Throughput versus parsing quality for open-source OCR models. Horizontal axis: pages per second (log scale). Vertical axis: equal-weight average score across four document parsing benchmarks.

Average score is the equal-weight mean of the four benchmarks. All models use accelerated configurations. Settings and reproduction: reproduction guide. Per-model throughput: model guide.

Benchmark comparisons#

Image: Radar chart comparing MiniCPM-OCR 1.0 with other OCR models across six benchmarks.

Full MiniCPM-OCR 1.0 evaluation results#

Image: Detailed OCR model results on OmniDocBench v1.6, Real5-OmniDocBench, Wild-OmniDocBench, olmOCR-Bench, CHAOS-Bench, and OCRBench v2.

Image: Per-benchmark bar charts for Real5-OmniDocBench, Wild-OmniDocBench, OmniDocBench v1.6, olmOCR-Bench, OCRBench v2, and CHAOS-Bench.

Each panel keeps its own benchmark's scores; higher is better. Scores from different benchmarks are not directly comparable. Test settings and reproduction steps are in the reproduction guide; per-benchmark discussion is in the model guide.

Throughput#

On an NVIDIA GeForce RTX 5090 with all 1651 OmniDocBench v1.6 images, image-level concurrency 1, and each system's accelerated configuration, MiniCPM-OCR 1.0 reaches 1.2 pages/s (relative throughput 1.00×; higher is faster).

Image: OCR system throughput with accelerated configurations; MiniCPM-OCR 1.0 achieves 1.2 pages/s.

Concurrency 1 is image-level: regions inside a page are still recognised in parallel. Real throughput also depends on page content, backend, concurrency, and hardware.

Examples#

Left to right: original document, detected layout, and rendered Markdown.

English documents

Image: English document animation: original, detected layout, and rendered Markdown.

Chinese documents

Image: Chinese document animation: original, detected layout, and rendered Markdown.

More examples#

Original documents and parsed results, side by side.

English documents

Image: English examples comparing inputs and results: handwriting, historical and scanned documents, newspapers, financial statements, and key information extraction.

Chinese documents

Image: Chinese examples comparing inputs and results: handwritten notes, complex formulas, newspapers, financial statements, stamped documents, illustrations, and key information extraction.

Model architecture#

Parsing pipeline#

Full-page parsing runs as a two-stage pipeline orchestrated by the minicpm-ocr SDK. The two stages use separate models, run in separate environments, and may run on different machines:

• Stage 1 — Layout detection (SDK environment). The PP-DocLayoutV3 layout model detects document regions — text, tables, formulas, charts, seals — and determines reading order; the SDK crops each region. It runs locally inside the SDK environment and requires PaddlePaddle / PaddleX.
• Stage 2 — Region recognition (inference service). The SDK sends each cropped region over HTTP to a MiniCPM-OCR 1.0 inference service — served by vLLM, SGLang, llama.cpp, or Ollama — with a prompt matching the region type, then assembles the recognized content into the final Markdown / JSON in reading order.

Image: Parsing pipeline: in the minicpm-ocr SDK environment, the PP-DocLayoutV3 layout model detects and crops document regions; each cropped region is sent over HTTP to a separate recognition service running MiniCPM-OCR 1.0 (vLLM, SGLang, llama.cpp, or Ollama); the SDK then assembles the recognized text, tables, formulas, charts, and seals into Markdown and JSON in reading order.

See the model guide for more on architecture, parsing workflow, deployment options, and benchmark breakdowns.

Quick start#

Use with the MiniCPM-OCR SDK#

Required before using either the CLI or Python API: deploy and start the OCR model service first, and wait until it is ready to accept requests. The SDK examples below are clients of that running service; they do not deploy or start the OCR model service automatically.

Use the official minicpm-ocr SDK to parse local images or PDFs with this GPTQ checkpoint. The SDK detects layout with PP-DocLayoutV3, recognises each region, and saves structured JSON and Markdown in reading order.

Install Python 3.10+ and the SDK:

pip install "minicpm-ocr[all]"

Install minicpm-ocr for your platform using the installation guide.

Prerequisite: deploy and start the model service#

Before running the SDK, start a vLLM recognition service for the GPTQ checkpoint in this repository, openbmb/MiniCPM-OCR-1.0-GPTQ, following the vLLM deployment guide. Use this repository ID as the served model path, --served-model-name minicpm-ocr, --port 8080, and --max-model-len 40960. The service reads the GPTQ quantization settings from this repository's config.json.

The SDK examples use http://127.0.0.1:8080, model alias minicpm-ocr, the vllm backend, and an NVIDIA GPU for layout detection. Match the URL, alias, and context length to your service. The SDK connects to the running service; it does not load the quantized weights into the SDK process.

CLI#

Start the model service described above and wait until it is ready before running this command. Save a local image as demo.png, then run:

minicpmvocr parse demo.png \
  --backend vllm \
  --server-url http://127.0.0.1:8080 \
  --model minicpm-ocr \
  --device cuda:0 \
  --set engine.max_model_len=40960 \
  --set run.run_root=output \
  --output output/demo.md

The CLI automatically downloads PP-DocLayoutV3 to models/PP-DocLayoutV3_infer/ when needed. The final Markdown is saved to output/demo.md; structured JSON and per-page Markdown are kept in output/<run_tag>/results_json/ and output/<run_tag>/output/.

Python API#

The model service described above must already be running before you execute this Python example. Set engine.server_url and engine.model_name to match that service.

The same pipeline is available from Python:

from importlib.resources import files
from pathlib import Path

from minicpmvocr import load_config, pipelined_run
from minicpmvocr.layouts.download import ensure_layout_model


def main():
    layout_model = ensure_layout_model(Path("models/PP-DocLayoutV3_infer"))

    cfg = load_config(
        str(files("minicpmvocr").joinpath("configs/minicpmvocr.yaml")),
        overrides={
            "input": {"file_path": str(Path("demo.png").resolve())},
            "engine": {
                "type": "vllm",
                "server_url": "http://127.0.0.1:8080",
                "model_name": "minicpm-ocr",
                "max_model_len": 40960,
            },
            "layout": {
                "model_dir": str(layout_model),
                "device": "cuda:0",
            },
            "pipeline": {"overlap_layout": True},
            "run": {"run_root": str(Path("output").resolve())},
        },
    )

    pipelined_run(cfg)

    print("JSON directory:", cfg["_meta"]["result_json_dir"])
    print("Markdown directory:", cfg["_meta"]["output_dir"])


if __name__ == "__main__":
    main()

ensure_layout_model() reuses or downloads the official layout model. Results are written to output/<run_tag>/results_json/ and output/<run_tag>/output/. To parse a PDF, replace demo.png with its local path.

For CPU layout detection, use --device cpu in the CLI or set layout.device to cpu in Python. This changes only the layout detector; the recognition service still uses the deployed quantized model.

Run minicpmvocr parse --help for CLI options, or see the SDK API reference for configuration and document-level Markdown export.

Prompts#

The prompt selects the task and must match the region you cropped:

If finish_reason is length, the output was truncated — raise the token budget.

Key information extraction (KIE)#

MiniCPM-OCR 1.0 can pull specified fields straight out of an image — for example the store name and amount on a receipt. Send the image with a field-extraction prompt and ask for a JSON object; you do not need to produce page Markdown first.

Use the image recognition API example and replace Text Recognition: with:

Extract the "store_name", "amount". Format the output as a JSON object with keys: "store_name", "amount". If an item is not in the image, use "###".

Replace both field lists with your own field names (Chinese or English). The response is model-generated text: parse the JSON and check fields and values before using it downstream.

Agent skill#

Have a coding agent follow the skill to set up the environment, download the model, and parse a first document:

Read and execute https://github.com/OpenBMB/MiniCPM-OCR/blob/main/docs/SKILL_en.md

Fine-tuning#

LLaMA-Factory · ms-swift — data preparation, full-parameter SFT with the multimodal projector frozen, training, and evaluation (French handwritten text lines).

Quantization#

Only the language-model linear layers are quantized. The vision encoder, multimodal projector, lm_head, and the GatedDeltaNet in_proj_a / in_proj_b projections stay in BF16, so image encoding is unchanged and recurrent gating is preserved.

Weight footprint drops from about 2.42 GiB (BF16) to about 1.74 GiB.

quantization_config in config.json is what vLLM reads; no extra flag is needed. quantize_config.json records the quantization parameters.

Related checkpoints#

DSpark requires a matching main and draft model from the same release — do not mix draft weights with a different main checkpoint. See DSpark acceleration.

For OCR quality assessment (auto-accept, human-review routing, best-of-N across models), see MiniCPM-OCR Jev.

FAQ#

Installation, model downloads, GPU memory, input formats, and unexpected outputs: FAQ.

License#

Model weights, code, and documentation are released under the Apache License 2.0.