tinybrainbot-100m-v5-base-ONNX

tinybrainbot-100m-v5-base (ONNX)#

This is an ONNX version of nkthebass/tinybrainbot-100m-v5-base. It was automatically converted and uploaded using this Hugging Face Space.

Usage with Transformers.js#

See the pipeline documentation for text-generation: https://huggingface.co/docs/transformers.js/api/pipelines#module_pipelines.TextGenerationPipeline

TinyBrainBot-100M-v5-Base#

A 100M-parameter base model trained from scratch on 29.6B tokens, as a plain Llama (LlamaForCausalLM): no custom code. It is TinyBrainBot-100M-v3-Base continued for 4.6B tokens with knowledge-dense data, and it beats it on every Open SLM Leaderboard task, without any benchmark training data.

Scores#

Zero-shot, full test sets, acc_norm; (raw acc) underneath. Intelligence Index from the Open SLM Leaderboard.

Other models' numbers are from the leaderboard.

Arithmetic it never trained on#

ArithMark-3 is kept honest: on 1,380 held-out arithmetic problems in topics and wordings the model never saw, it scores 40.7, close to its public 47.0.

Quick start#

from transformers import AutoModelForCausalLM, AutoTokenizer
tok = AutoTokenizer.from_pretrained("nkthebass/tinybrainbot-100m-v5-base")
model = AutoModelForCausalLM.from_pretrained("nkthebass/tinybrainbot-100m-v5-base")
ids = tok("The water cycle begins when the sun heats water in oceans and lakes, causing it to", return_tensors="pt")
print(tok.decode(model.generate(**ids, max_new_tokens=24, do_sample=False)[0]))

GGUF files for llama.cpp, LM Studio, Ollama and Jan: F16 (201 MB), Q8_0 (107 MB) and Q4_K_M (66 MB), with add_space_prefix=false set so they tokenize exactly like transformers (greedy output matches).

This is a base model: it continues text and does not chat.

Model#

Training#

29.6B tokens: the 25.06B of 100M-v3-Base, then 4.56B tokens in five stages, each re-warming the learning rate (peak 3e-4 to 6e-4) and decaying it again. The published weights are the final step of the last stage (step 113,000), not a merge.

Grade-school science Q&A. The final stage includes about 420k grade-school science questions with short explained answers, written by Ling-3.0-Flash: no answer options, and any item sharing an 8-word phrase with an ARC question was removed before training.

No benchmark train or test data was used in the continuation.

Evaluation#

lm_eval --model hf --model_args pretrained=nkthebass/tinybrainbot-100m-v5-base,dtype=float32 --tasks hellaswag,arc_easy,arc_challenge,piqa --num_fewshot 0
python bencharithmark-3.py --model nkthebass/tinybrainbot-100m-v5-base --dtype float32

Limitations#

A 100M base model: good at short factual and commonsense continuations for its size, but open-ended generation is limited and can be wrong. Not aligned or safety-tuned. An instruct version is planned.