reflexbench
Open benchmark for System One decision models on Choice, Score and Noul, reporting accuracy, calibration and robustness separately.
Overview
ReflexBench is a provider-neutral benchmark and evaluation harness for System One and typed decision models such as TypeSafe Jev, Laya, Reflex/Qwen, Kev, jeff and Verdict. Version 1.0.0 freezes corpora, manifests and hashes and reports several dimensions separately rather than one best-model score: semantic accuracy and macro-F1, NLL, Brier and ECE, score error, option-order stability, multilingual consistency, Choice cardinality, failures, latency boundaries, and paired bootstrap intervals.
The headline leaderboard uses a frozen 111-case public-hard cohort: TypeSafe Jev 73.0% raw semantic accuracy, upstream Reflex/Qwen3.5-2B 41.4%, a frozen Qwen3.5-0.8B readout 39.6%, jeff 37.8%, Verdict 1.4 36.9%, Laya base 35.1% and Kev-0.8B 32.4%. A separate research lane measures how much a deterministic Reflex Core policy adds on top of the same model response, for example Jev 96.4% to 100% on one 110-case set with zero extra calls. The authors stress these are their own same-corpus measurements, not a universal ranking.
What the decision model decides
| Type | Question | Answers |
|---|---|---|
| Choice | Benchmark Choice items with fixed option sets | — |
| Score | Benchmark Score items on ordered rubrics | — |
| Yes / no | Benchmark Noul/binary items | — |
How it works
- Freeze the corpus and record engine identity and revision
- Run each engine through an adapter (e.g. a /v1/systemone endpoint)
- Score accuracy, calibration, robustness and failures separately
- Optionally apply deterministic Reflex Core policy to the same responses and compare
Strengths
- Reports calibration and robustness, not just accuracy
- Provider-neutral adapter contract with a submission process
- Failures kept in the evidence ledger
Caveats
- Small public-hard cohort (111 cases)
- Run by Brida, which also builds a Reflex consumer
- Third-party datasets not always redistributed
Getting started
PYTHONPATH=reflexbench python reflexbench/systemone_reflex_runner.py --help against a System One-compatible endpoint.At a glance
- Type
- Benchmark
- Status
- production
- Language
- Python
- Runtime
- CLI harness
- Has code
- yes
- License
- Apache-2.0
- Stars
- 6
Use cases and integrations
How curated lists describe it
ReflexBench — open benchmark and evaluation harness for System One models and typed decision engines
wh000wh000/awesome-jev-liveReflexBench — open benchmark and evaluation harness for System One models and typed decision engines.
hellogumbo/awesome-jev
Frequently asked questions
Which model scored highest?
On the 111-case public-hard cohort, TypeSafe Jev at 73.0% raw semantic accuracy, per the README.
Does it give one overall score?
No; accuracy, calibration, robustness and latency are reported separately.
Can I submit my own model?
Yes, SUBMIT_A_MODEL.md describes the adapter contract and result workflow.
Related projects
pytest-style tool that pins fixtures and expected Jev answers, then fails a model upgrade when answers flip or confidence drops.
ProjectbetaPythonWrite-up testing whether Jev beats small fast LLM tiers on public and private classification, as a classifier or a filter.
ProjectresearchPythonHarness that runs Jev on all 19,646 TMMLU+ Traditional Chinese questions as four-way Choices, reporting accuracy and calibration.
ProjectresearchPythonReliability testing for typed decision models: contracts, calibration metrics, metamorphic fuzzing and CI pass/warn/fail gates.
DemoprototypePythonPre-registered audits of AI judges that measure how many decisions a judge can take alone at a given error rate, with calibration.
BenchmarkbetaPythonClaude Code plugin and CLI for the Jev funnel: judge items in bulk, settle confident ones in code, send only uncertain ones onward.
PluginbetaPython