kev-0.8b

Kev-0.8B#

Model summary#

Kev-0.8B is a decision model. It reads one document (the state) and a set of typed questions about it, and returns a calibrated probability distribution over the options supplied with each question, in a single forward pass and without generating text. It implements TypeSafe's public System One API (POST /v1/systemone), so the TypeSafe SDK works against it unchanged. It is the smallest Kev: a LoRA adapter and a pointer head on Qwen3.5-0.8B-Base that runs on a laptop or a 4 GB GPU. It is intended where memory or cost rules out the larger sizes and the task resembles what it was trained on; out of domain it is markedly less accurate than Kev-4B. This card describes the checkpoint in Kev 1.0, first published on 2026-09-24.

Model details#

Input. A state (text, or a JSON object or array rendered as labelled text) and any number of named questions, each of one of three types:

Each question is answered as its own row that continues from the shared state, so questions cannot influence one another; the state is computed once and cached.

Intended uses#

• Typed decisions close to its training families (document classification, routing, policy checks over stated rules) on hardware too small for Kev-4B: a laptop, an L4 or a 4 GB GPU.
• A starting point for fine-tuning on the user's own labels where training cost matters (kev.train --init_from jaredpalmer/kev-0.8b).
• Prototyping against the System One API before moving to a larger Kev.

Out-of-scope uses#

• Text generation, chat, summarisation or open-ended question answering. The model only scores the options it is given.
• Tool-call routing (whether to call a tool, ask for a missing parameter or decline): its When2Call accuracy is below chance (see Limitations).
• Fully automated decisions with legal, medical, financial, employment or similar consequences for people, without human review.
• Knowledge-heavy questions, date arithmetic, states longer than 65,536 tokens, and languages other than English.

How to use#

Serve it with the Kev repository. On CUDA it runs in bf16 with fused DeltaNet kernels and CUDA graphs (an L4 is enough); on Apple Silicon the same command serves it through MLX, chosen automatically.

git clone https://github.com/jaredpalmer/kev.git && cd kev && uv sync --extra serve
uv run --extra serve python -m kev.serve --run jaredpalmer/kev-0.8b --port 8008        # Kev 1.0 (this card)
uv run --extra serve python -m kev.serve --run jaredpalmer/kev-0.8b@v1.0 --port 8008   # the same weights, pinned

from typesafe_sdk import Choice, Noul, TypeSafeClient

client = TypeSafeClient(api_key="local", base_url="http://127.0.0.1:8008", model="kev-latest")
response = client.system_one(
    state="I was charged twice for order 1182. Please refund one of the charges.",
    questions={
        "team": Choice(instructions="Which team should handle this?",
                       criteria={"billing": "Charges and refunds", "shipping": "Deliveries", "returns": "Exchanges"}),
        "urgent": Noul(instructions="Does this need a reply today?"),
    },
)
print(response.choices["team"].choice, response.nouls["urgent"].noul)

The calibrated temperature is applied by default; KEV_TEMPERATURE=1.0 returns the raw probabilities. KEV_DTYPE=fp32 selects the exact path used for evaluation. A state over 65,536 tokens is refused with a 422 that gives its token count.

Training data#

The two fine-tuning stages replay 2,000 and 6,000 records from decision-v7. No output of Jev (TypeSafe's hosted decision model) was used. CodeReviewer and FlakeFlagger come from Zenodo; the CFPB narratives are US government works; per-source licences and revisions are recorded in the suite manifests. The evaluation-only suites below (breadth-v1, tasksource-heldout-v1, transfer-v4, longdoc-v1, and the When2Call and prompt-injection sources of devtools-v1) never enter training.

Training procedure#

1. Base recipe. Two epochs on decision-v7 from the base: LoRA rank 16, α 32; learning rate 1e-4, one-cycle schedule; batch 8; bf16 autocast; seed 2. The loss is cross-entropy over each question's options. Option order is shuffled, "none of the above" options and distractors are inserted at random, and a quarter of choice records also yield a minimal pair (the question with a "none of the above" option, once with the correct option present and once with it removed).
2. Dates and missing evidence. One epoch from stage 1 at learning rate 4e-5 with 2,000 replayed records.
3. Documents and skills. One epoch from stage 2 on all three training sets together at learning rate 2e-5 with 6,000 replayed records; batch 4 × 2 accumulation; states of at most 7,552 tokens; gradient checkpointing; seed 1; 2,818 optimizer steps.
4. Calibration. A single temperature, T = 2.35, minimising negative log-likelihood on the stage-3 trial's decision-v7 development rows (1,264 questions). These are held-out items of a training corpus. A refit on held-out datasets was evaluated and not adopted (see Calibration).

Evaluation#

Methodology. Every number is the fp32 evaluation path at the shipped temperature unless stated. Development partitions were used for selection; test partitions were read once for this checkpoint; the transfer-v4 test is locked (read once per candidate) and was judged against a bar fixed in advance. Paired intervals are 95 % bootstraps that resample whole records (2,000 resamples), so questions sharing a state move together. Differences are in percentage points (pp). Jev (TypeSafe's hosted model, queried through Vercel AI Gateway) is shown where it was read on the same items. The suites:

• breadth-v1: 14 held-out public datasets in five areas (knowledge, language, retrieval, tools, arts), never trained on.
• tasksource-heldout-v1: 24 whole task families of a public multi-task collection, never trained on (family names private).
• transfer-v4: out-of-domain decisions from six never-trained public sources (QNLI, SciQ, TweetEval-offensive, PAWS, MMLU, Emotion) plus held-out policy and rule structures.
• hard-v1: the skill families above; the test split holds out templates of trained generators.
• devtools-v1: developer-tooling decisions from six licence-checked sources (four trained, two evaluation-only).
• documents-v1 / documents-v2: CFPB complaint narratives; v2 is a private held-out test set.
• longdoc-v1: CUAD commercial contracts and generated agreement bundles, with states of 4k to 64k tokens.

Headline panels marked "audited" exclude items that a label audit found unsound¹; every exclusion removes the same rows from both sides of a comparison.

Held-out data (never trained on).

Trained families (held-out items and templates).

Jev's devtools-v1 figure is over all 1,074 development questions; Kev's rows drop a CodeReviewer id that the suite's builder reused for two records (2 questions).

Against the previous version (tag night2-du-release, at its own temperature 2.41; registered criteria, each test read once):

Long documents.

• Validated context length: 8,192 tokens, the trained length. The 16k bucket is outside the tolerance: its lower bound is −8.5 pp, below −3 pp, so no longer length is validated.
• Rule, fixed before the read: the validated length is the nominal size of the largest bucket from 16,384 tokens up such that it, and every bucket between it and 8,192, is within tolerance. Within tolerance means the CUAD accuracy difference from the 8k bucket (states of 6,553–7,618 tokens, the trained length), paired on the same contract, repeat and question, has a 95 % lower bound of at least −3 pp, and every record was answered. If the 16k bucket fails, the validated length is 8,192 tokens.

CUAD accuracy, ECE and the paired difference from the 8k bucket by nominal state length (longdoc-v1 development):

ECE at the shipped T = 2.35. Δ is paired on the 445–447 questions asked about the same contracts at both lengths. The 4k bucket holds different contracts and is not a reference for the rule. Source: runs/r28-readout/context.json (round 28's registered read-out, runs/r28-08b-r15-longdoc).

Calibration (expected calibration error, ECE, at the shipped T = 2.35; lower is better):

The shipped temperature was fitted on held-out items of a training corpus, which the project's rules no longer allow for a new release. A registered refit on 648 questions from held-out datasets (the calibration split of transfer-r3, eight sources, and 200 MMLU-Pro questions) gives T = 2.52 (90 % bootstrap interval [2.19, 2.83]). On the 4,468 audited breadth-v1 and tasksource-heldout-v1 development questions it is better calibrated: ECE 0.037 against 0.048, Brier lower by 0.0020 [0.0014, 0.0025]. It is worse on the trained families, each by more than the registered tolerance of 0.005: hard-v1 ECE 0.124 against 0.112, devtools-v1 (audited) 0.099 against 0.092, documents-v1 0.078 against 0.059. The refit therefore did not qualify, and T = 2.35 stays. Users whose workload resembles held-out public datasets more than Kev's training families can serve it at the refit value with KEV_TEMPERATURE=2.52; answers do not depend on T.

Other results.

Serving. CUDA, bf16 with fused kernels and CUDA graphs, on an L4; model time per request (median of 20) for a new / repeated state: 22.7 / 16.1 ms for six questions about a short state, 108.6 / 32.3 ms for five questions about a 2,200-token state; 62.8 requests/s at 64 clients. Resident GPU memory is 3.8 GB. Served probabilities stay within 0.017 of the fp32 evaluation path on 280 questions, with 1 changed answer.

Apple Silicon (MLX, bf16, M5 with 32 GB; three questions, one about a fact planted at 60 % depth; the state is prefilled in 1,024-token chunks):

Against fp32 PyTorch on the CPU on the same requests, the MLX answers differ by at most 0.0076 at 8k and 0.0052 at 16k tokens, with no changed answers. On 100 short-state questions the MLX path is within 0.020 of the fp32 evaluation path, with 1 changed answer.

¹ Excluded from audited panels: four breadth-v1 datasets (routerbench, whose states lack the information asked for; cfcolor and humicroedit, at chance for every system; chessbench, at the floor for every system); seven tasksource-heldout-v1 families with invalid or unrecoverable labels (names private); two devtools-v1 tasks whose labels the state does not determine (flakeflagger, commit change type).

² The community Decision Index 0.2's chance-corrected index: per dataset (score − chance) / (1 − chance), averaged within each area, then 100 × the mean of the five areas. Jev's index is from a separate read of the same test items.

Limitations and trade-offs#

• It is a sub-1B model out of domain. It trails Jev by 21 points on transfer-v4 development and by 31 points on the breadth-v1 index; knowledge (MMLU-Pro 0.230) and paraphrase are near the untrained base. Use Kev-4B where accuracy matters.
• Its gains are in distribution. The training splits of documents-v1, hard-v1 and devtools-v1 are in its training data; on JevBench's public hard tier, an out-of-distribution check, the documents-and-skills stage moved it +2.7 pp [−1.8, +7.2], not distinguishable from zero.
• Tool-call routing got worse. When2Call, an evaluation-only source, fell from 0.260 to 0.167 on development and from 0.233 to 0.133 on test, below the one-in-four rate of guessing among its four options. Do not use it for tool-call routing.
• One devtools-v1 result is unexplained. FlakeFlagger moved 0.500 → 0.520 on development but 0.507 → 0.813 on test, on 150 questions per split; treat it as unexplained, not as a skill.
• Date arithmetic is its weakest family: 0.35 on the deadline policy questions against Jev's 0.95; the KEV_DATE_FACTS=1 preprocessor helps the larger models more than this one.
• Rule composition is weak: both siblings of a held-out policy minimal pair are right 0.422 of the time.
• Calibration is one in-distribution temperature (see Calibration). It cannot reorder confidences: coverage at ≤ 5 % error out of domain is 0.145 on development against Jev's 0.70, so few decisions can be automated at a strict error budget.
• Untrained lengths. Training states were at most 7,552 tokens. Longer states are served up to 65,536 tokens; how far accuracy holds is the validated context length above.
• Option order can change an answer; question isolation does not prevent this.

Bias, risks and ethical considerations#

• Calibrated probabilities can create unwarranted trust. The temperature was fitted on development rows of the training distribution and does not transfer to every workload; measure accuracy and calibration on a labelled sample of your own data, and refit the temperature there (python -m kev.calibrate), before setting thresholds.
• Accuracy and calibration shift under domain change, and more at this size than at larger ones. Monitor production error rates rather than relying on the numbers above.
• Do not use it for consequential automated decisions about people without human review. Biases of the base model and of the training data (including labels produced by other models) are not measured.
• States may contain personal or confidential data. Self-hosting keeps inputs on your own hardware; the server is open unless KEV_API_KEY is set, so apply your own access control and data-handling policy.

Compute#

• Base recipe: about 20 minutes on one NVIDIA H100. Dates stage: 9 minutes.
• Documents and skills stage: 52 minutes on one NVIDIA H200 (peak 23.9 GB).
• Evaluation and serving checks: single H100 / H200 / L4 GPUs on Modal; MLX measurements on an Apple M5.

Provenance and reproducibility#

• Code, suites and evaluation reports: github.com/jaredpalmer/kev. Release numbers: runs/release/kev-08b-r15.json (scripts/release_numbers.py --release kev-08b-r15), the locked read runs/locked/kev-08b-r15-ungated/, the 2026-09-30 family reads runs/fam-08b-breadth/, runs/fam-08b-breadthtest/ and runs/fam-breadth-test-report/, the calibration refit runs/r28-readout/round28.json, serving runs/serve-08b-l4/, runs/mlx-long-states/, runs/mlx-full-0.8b/.
• Stages: base trial q35-08b/02-trial-2 (tag v7-base); dates night2-08b-du2/00-trial-0 (tag night2-du-release); documents and skills round 15 r15-08b/00-trial-0 (experiments/round15/joint.json, rule experiments/rounds/r15.json). Calibration refit: round 28 arm 08b-r15 (experiments/rounds/r28.json).
• Released weights: Hub revision 9a45d25e; adapter sha256 9b908623…, head.pt sha256 f400bd12… (T = 2.3511).
• Release history: published 2026-09-24 as round 15's confirmed candidate; included unchanged in Kev 1.0. The record of how it was selected, including suites since retired as unsound (scienthoon, WANLI-v2, TypeSafe), is the README at Hub revision 9a45d25e and PLAN.md at git tag research-archive-2026-09-24.

Citation#

@misc{palmer2026kev08b,
  title        = {Kev-0.8B: a calibrated decision model on Qwen3.5-0.8B},
  author       = {Palmer, Jared},
  year         = {2026},
  howpublished = {\url{https://huggingface.co/jaredpalmer/kev-0.8b}},
  note         = {Kev 1.0}
}

Contact#

Questions and issues: github.com/jaredpalmer/kev/issues.