UltraX-0.6B-Preview

UltraX: Refining Pre-Training Data at Scale with Adaptive Programmatic Editing#

OpenBMB UltraData

UltraX Logo

📜 Paper | 💻 Code | 🤗 Datasets | 📦 UltraData Collection

English | 中文

📚 Introduction#

UltraX is a function-calling refinement framework for large-scale pre-training data. Instead of end-to-end text rewriting, UltraX trains a lightweight refinement model to predict structured editing operations — including insertion, deletion, and modification — which are then deterministically executed on the original text.

This repository provides the UltraX refinement models, which are trained via full-parameter SFT to predict cleaning function calls for web-crawled text:

📢 News#

• [2026.07.13] UltraX codebase, refinement model, and refined datasets are now available on GitHub and Hugging Face. 🚀🚀🚀
• [2026.07.10] UltraX technical report is available on arXiv. 🔥🔥🔥

💡 Highlights#

• Function-Calling Refinement: Instead of end-to-end text rewriting, UltraX predicts structured editing operations (keep_all, remove_all, remove_lines, replace_str, add_line), enabling fine-grained instance-level editing with deterministic execution.
• LAM + DCR Pipeline: Line Alignment and Mapping (LAM) aligns original and refined text at line level, while Dynamic Context Replacement (DCR) converts character-level edits into reliable replace_str operations with unique context anchoring.
• Lightweight & Efficient: The refinement model is a standard SFT model that predicts structured function calls, significantly cheaper than end-to-end LLM rewriting while achieving superior performance.

🔬 Pipeline Overview#

Image: image

🔍 Model Details#

All models are trained with the following configuration:

Model Variants#

• UltraX (Instruction-Guided, Edit-Weighted): The default configuration. The model is trained with a system instruction that defines the editing operations and cleaning rules, using edit-weighted sampling.
• UltraX-No-Instruction: Trained without system instruction. Demonstrates that programmatic refinement brings clear gains even without explicit task protocol (Avg: 45.73 vs Raw 45.08), though instruction guidance further improves performance (Avg: 46.14).
• UltraX-Preservation-Weighted: Trained with preservation-weighted sampling where keep_all accounts for 60% of training data. Shows stronger performance on certain tasks (ARC-C, ARC-E, OBQA, SciQ), demonstrating the trade-off between editing aggressiveness and content preservation.

📈 Evaluation Results#

Performance comparison on FineWeb (20B tokens, 1B MiniCPM, 10 benchmarks, zero-shot):

Image: image

Image: FineWeb Token Curve Average downstream performance on FineWeb under different training token budgets.

🔧 Function Space#

The model predicts operations from the following function space:

Input/Output Format#

Input: Text with line number markers (<lid:N> prefix per line), preceded by a system instruction.

Output: A sequence of function calls, one per line. Example:

remove_lines(1, 3)
replace_str(5, 'Click here to subscribe', '')
remove_lines(12, 15)

🚀 Quick Start#

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("openbmb/UltraX-0.6B-Preview")
tokenizer = AutoTokenizer.from_pretrained("openbmb/UltraX-0.6B-Preview")

❤️ Acknowledgements#

• The refinement model is trained using ms-swift.
• UltraX builds upon insights from ProX and RefineX.

Thanks for their awesome work! Open-source contributions make UltraX possible! 🙌

📖 Citation#

If you find our work useful, please consider citing:

@misc{ultrax2026,
  title={UltraX: Refining Pre-Training Data at Scale with Adaptive Programmatic Editing},
  author={Xinlong Zhao and Dongsheng Liu and Hengyu Zhao and Zixuan Fu and Zheng Wang and Jie Cai and Jie Zhou and Qiang Ma and Xuanhe Zhou and Xu Han and Yudong Wang and Zhiyuan Liu},
  year={2026},
  eprint={2607.08646},
  archivePrefix={arXiv},
  primaryClass={cs.CL},
}

📜 License#

This project is licensed under the Apache 2.0 license.