S1MB
System One Mosaic Benchmark comparing Jev and open decision models across 137 English and 27 Japanese Choice, Noul and Score tasks.
Overview
S1MB (System One Mosaic Benchmark) is a shared reference for comparing System One decision models on the same inputs and metrics. Models select an option (Choice), make a yes/no judgment (Noul) or assign a score from supplied criteria (Score). The English v1 scope has 137 benchmarks across 106 dataset subsets, including six synthetic probes of how models respond to varied instructions, contexts and criteria; Japanese v1 has 27 benchmarks and 8,067 judgments, with that dataset kept private.
The mosaic combines tasks from public NLP datasets with synthetic tasks in one workflow, preserving structured inputs, authored criteria and soft targets. It reports raw task metrics (for example Brier score for Noul) alongside baseline-adjusted scores and relative rankings, with missing measurements left visible. Results are published as a dataset with an interactive Hugging Face leaderboard, adapters exist for TypeSafe/Jev and other models, and anyone can submit results. The authors note scores do not establish unseen-task generalisation or training-data non-overlap.
What the decision model decides
| Type | Question | Answers |
|---|---|---|
| Choice | Benchmark Choice tasks drawn from public NLP datasets | — |
| Yes / no | Benchmark Noul tasks scored by Brier against the authored true category | — |
| Score | Benchmark Score tasks with supplied criteria | — |
How it works
- Choose a scope (english-v1, japanese-v1 or both)
- Run s1mb with a model adapter
- Compute raw metrics and baseline-adjusted scores
- Publish to the results dataset and leaderboard
Strengths
- Broad coverage across 100+ tasks
- Raw and adjusted scores both visible
- Open submission process
Caveats
- Does not prove generalisation or training-data non-overlap
- Japanese evaluation data is private
Getting started
Follow docs/quickstart.md, then run s1mb run --category english-v1.At a glance
- Type
- Benchmark
- Status
- production
- Language
- Python
- Runtime
- CLI harness
- Has code
- yes
- License
- MIT
Use cases and integrations
How curated lists describe it
Decision-model evaluation: compares Jev and open decision models across 137 English Choice, Noul, and Score benchmarks, including six synthetic generalization probes, with public evaluation data, recorded results, and an interactive [leaderboard](https://huggingface.co/spaces/hotchpotch/S1MB-leaderboard).
yibie/awesome-jevCompares Jev and open decision models across 137 English Choice, Noul, and Score benchmarks, including six synthetic generalization probes, with public evaluation data, recorded results, and an interactive [leaderboard](https://huggingface.co/spaces/hotchpotch/S1MB-leaderboard).
cobanov/awesome-jev
Frequently asked questions
What does S1MB measure?
How decision models perform on Choice, Noul and Score tasks across 137 English and 27 Japanese benchmarks.
Can I add my own model?
Yes, implement the typed adapter contract and submit results.
Does a high score mean a model generalises?
Not necessarily; the authors say it does not establish unseen-task generalisation.
Related projects
Evaluation suite for decision models with 23,900 pinned rows across nine task families and three primitives, plus a leaderboard.
ProjectbetaPythonOpen-source BYOK arena where Jev faces another model as a judge on your question, with blind voting before costs are revealed.
ProjectbetaTypeScriptpytest-style tool that pins fixtures and expected Jev answers, then fails a model upgrade when answers flip or confidence drops.
ProjectbetaPythonBenchmark harness comparing Jev with OpenRouter models and open Jev-shaped models on accuracy, calibration, latency and cost.
ProjectresearchPythonWrite-up testing whether Jev beats small fast LLM tiers on public and private classification, as a classifier or a filter.
ProjectresearchPythonAudit of 300 web pages for AI citability with nine Jev decisions each, checked against an LLM judge for agreement, cost and latency.
ProjectresearchPython