JevBench
MIT-licensed benchmark and leaderboard for Jev-class decision models: accuracy, calibration, latency and cost on sealed tiers.
Overview
JevBench by Benchmark Heaven measures typed decision models, including TypeSafe's Jev and Jev-compatible systems, on typed decision accuracy, probability calibration, latency and cost, independently of TypeSafe. Listings describe 534 English decisions split into public and sealed tiers; only aggregate statistics are published for sealed items, and a large gap between public and sealed accuracy reduces the Intelligence axis to discourage training on the public half.
The current release (v1.6.1, published 2026-10-06 with addenda) headlines a Capability Score, the mean of Intelligence and Calibration within cost and latency caps, on an open-weights board, and a Composite Score (a weighted harmonic mean of Intelligence, Calibration, Speed and Cost with penalties) leading a hosted API board. Historical releases are preserved with their own methods: in v1.4.2.2, Imajev-4B ranked first at 67.37 and Jev 1.13.0 fourth at 63.29 among 91 ranked systems, with Jev noted as stronger on raw intelligence and calibration than some faster, cheaper models. Combination experiments such as cascades and committees are reported separately. Published bytes are pinned by SHA-256, and the harness has CI tests.
What the decision model decides
| Type | Question | Answers |
|---|---|---|
| Other | English typed decisions (Noul, Choice, Score) scored for accuracy and calibration | — |
How it works
- Run a model on public and sealed decision tiers
- Score Intelligence, Calibration, Speed and Cost
- Combine into Capability and Composite scores with caps and penalties
- Publish versioned boards and aggregate artifacts
Strengths
- Independent, versioned and hash-pinned results
- Sealed tier limits overfitting
- Covers open-weights and hosted APIs
Caveats
- Scoring definitions change between versions
- Cost for open models uses reference prices, not GPU bills
At a glance
- Type
- Benchmark
- Status
- production
- Language
- Python
- Runtime
- Benchmark harness
- Has code
- yes
- Has tests
- yes
- License
- MIT
- Stars
- 268
Use cases and integrations
How curated lists describe it
MIT-licensed harness and leaderboard of 534 English decisions with public and sealed tiers, reporting accuracy, latency, and price together. Discussion: [Show HN](https://news.ycombinator.com/item?id=49800574).
AnotiaWang/awesome-decision-modelsJevBench v1 - a benchmark for Jev-class typed decision models: smart, cheap, fast, reliable, open.
logicrw/awesome-jev-projectsJevBench by Benchmark Heaven: benchmark and leaderboard for AI decision models, measuring typed decision accuracy, calibration, latency and cost.
wh000wh000/awesome-jev-livePublic evaluation of Jev and Jev-class typed-decision systems, with versioned aggregate results and measurement conditions.
cobanov/awesome-jevJevBench v1 - a benchmark for Jev-class typed decision models: smart, cheap, fast, reliable, open.
hellogumbo/awesome-jevBenchmark for typed decision models across several suites, with confidence cascades and committees reported separately.
MrJev/awesome-jev
Frequently asked questions
What does JevBench measure?
Typed decision accuracy, calibration, latency and cost.
How did Jev 1.13.0 rank?
Fourth at 63.29 in the historical v1.4.2.2 release; current boards are on benchmarkheaven.com.
Can I compare scores across versions?
The README warns not to, since methods differ between versions.
Related projects
Probability-aware evaluation of decision models: calibration, coverage at a fixed error budget and latency, Jev vs GLiNER2.5.
BenchmarkresearchPythonReproducible benchmark comparing Jev with hosted and open decision models on rule application, probabilities, latency and cost.
BenchmarkresearchPythonMetamorphic benchmark testing whether decision-model probabilities stay coherent under 50 laws, with no gold labels.
BenchmarkresearchPythonBenchmark of nine local Jev-style decision engines on Apple Silicon across English and Korean sets, with paired significance tests.
BenchmarkresearchPythonFirst Chinese calibration benchmark for Jev-class models: accuracy, ECE, option order and script robustness.
BenchmarkresearchPythonPre-registered field test of Jev, GLiNER2.5-Decide, Laya and other decision models against small LLMs on fresh September 2026 data.
ProjectresearchPython