sysone-bench
Independent head-to-head benchmark of System One decision models on one sealed manifest, from Jev and Laya to 49 open-weight entries.
Overview
sysone-bench is an independent benchmark of System One decision models. Its sealed v2.0.0 release runs Laya (open weights), Jev (closed API) and Qwen with parallel constrained decoding on the same 1,190 cases and 1,550 typed questions in one run, from one sealed manifest and seed, with inputs verified byte-identical before comparison. The author's motivation is that earlier cross-vendor claims used different prompts and could not be compared.
The panel has since expanded to the Decision Index 0.2.1 set, with 49 distinct open-weight measurements scored on corrected gold; a script derives counts from verified run directories and collapses repeat or identical runs. Jev 1.13.0, as the closed reference, scores 0.9065 accuracy on 1,240 evaluation decisions; the best open model, kev-4b (4.66B, LoRA plus head, on one T4), scores 0.8556, followed closely by several 4B-9B models. Jev led Laya on all nine suites, with eight differences surviving Holm correction. The Jev key is passed on stdin and never touches disk. Code is MIT.
What the decision model decides
| Type | Question | Answers |
|---|---|---|
| Choice | Benchmark Choice questions across suites | — |
| Yes / no | Benchmark Noul questions | — |
| Score | Benchmark Score questions | — |
How it works
- Seal a manifest of cases and questions
- Run each model on byte-identical inputs
- Verify artifacts by checksum and deduplicate runs
- Compare accuracy with paired significance tests
Strengths
- Same bytes for every model
- Large open-weight panel
- Careful deduplication and verification
Caveats
- Accuracy-focused; other dimensions less emphasised
- Open-weight panel excludes closed models other than Jev
Getting started
See ops/ for runners; pipe the key on stdin: printf '%s' "$TYPESAFE_API_KEY" | .venv/bin/python ops/remote/remote_jev.py.At a glance
- Type
- Benchmark
- Status
- research
- Language
- Python
- Runtime
- Benchmark scripts
- Has code
- yes
- License
- MIT
Use cases and integrations
How curated lists describe it
First independent head-to-head benchmark of System One decision models.
wh000wh000/awesome-jev-liveFirst independent head-to-head benchmark of System One decision models (Laya vs Jev) on byte-identical inputs.
hellogumbo/awesome-jev
Frequently asked questions
How does Jev compare to open models?
Jev 1.13.0 scored 0.9065 versus 0.8556 for the best open model, kev-4b.
Why is it trustworthy?
All models get byte-identical inputs from one sealed manifest, and runs are checksum-verified.
How many models were measured?
49 distinct open-weight measurements plus the Jev reference.
Related projects
Evaluation suite for decision models with 23,900 pinned rows across nine task families and three primitives, plus a leaderboard.
ProjectbetaPythonJev-compatible open System One engine with a Claude Code-style terminal REPL showing live probabilities, plus an experiment harness.
ProjectprototypePythonLlamaIndex notebook comparing Jev with open decoders, encoders and specialised tools on document-pipeline Choice tasks.
ProjectresearchPythonBenchmark harness comparing Jev with OpenRouter models and open Jev-shaped models on accuracy, calibration, latency and cost.
ProjectresearchPythonReproducible measurements of Jev's real cost, latency, response shapes and accuracy, plus 13 decision models on the same 633 items.
ServiceresearchShort report comparing Jev and many other fast decision classifiers with an LLM on grading section relevance in technical documents.
Projectresearch