jev-benchmarks
Probability-aware evaluation of decision models: calibration, coverage at a fixed error budget and latency, Jev vs GLiNER2.5.
Overview
jev-benchmarks evaluates typed decision models on more than whether they choose the right label: whether their probabilities are calibrated enough to support automation, how much work can be accepted at a fixed error budget, what resources each decision uses, and end-to-end latency. Its first study compares Jev with fastino/gliner2.5-multi-v1 on zero-shot single-label classification using three BTZSC conditions with fixed model and data revisions, identical examples and label descriptions, a uniform negative control and paired target-stratified bootstrap intervals.
On a 300-example pilot, Jev scored 0.910 vs 0.700 on AG News and 0.870 vs 0.610 on Banking77/BTZSC, with coverage at 5% or lower error of 0.830 and 0.860 versus 0.240 and 0.270 for GLiNER. On DAIR Emotion the accuracy difference was unresolved (0.480 vs 0.440) and Jev was much worse calibrated (Brier 0.846 vs 0.668, NLL 5.588 vs 1.381, zero probability on the true label for 16% of examples), giving zero coverage. GLiNER ran locally and was faster on small label sets; Jev was slightly faster on the 72-label task. It has CI and an Apache-2.0 licence.
What the decision model decides
| Type | Question | Answers |
|---|---|---|
| Choice | Which label best describes this text? | — |
How it works
- Fix model and dataset revisions
- Run both models on identical examples and label descriptions
- Compute accuracy, Brier, NLL and coverage at error budgets
- Report paired bootstrap intervals and latency
Strengths
- Focuses on calibration and selective automation
- Rigorous paired statistics
- Reports mixed results honestly
Caveats
- Pilot of 300 examples
- Latency comparisons are deployment-specific
At a glance
- Type
- Benchmark
- Status
- research
- Language
- Python
- Runtime
- benchmark scripts
- Has code
- yes
- Has tests
- yes
- License
- Apache-2.0
Use cases and integrations
How curated lists describe it
A benchmark comparing Jev and GLiNER on text classification, probability calibration and selective automation.
logicrw/awesome-jev-projectsReproducible evaluation for calibration, selective risk, and latency.
cobanov/awesome-jevProbability-aware evaluation for typed decision models: calibration, selective risk, latency, and reproducible benchmarks.
hellogumbo/awesome-jevProbability-aware evaluation: calibration, and how much work can be automated at a fixed error budget.
MrJev/awesome-jev
Frequently asked questions
What does coverage at a fixed error budget mean?
How much of the data can be automated while keeping error at or below, say, 5%.
Where did Jev struggle?
On DAIR Emotion, where its calibration was much worse than GLiNER's.
Was Jev faster?
Only on the 72-label task; GLiNER running locally was faster on small label sets.
Related projects
- JevBench★ 268
MIT-licensed benchmark and leaderboard for Jev-class decision models: accuracy, calibration, latency and cost on sealed tiers.
BenchmarkproductionPython Reproducible benchmark comparing Jev with hosted and open decision models on rule application, probabilities, latency and cost.
BenchmarkresearchPythonMetamorphic benchmark testing whether decision-model probabilities stay coherent under 50 laws, with no gold labels.
BenchmarkresearchPythonBenchmark of nine local Jev-style decision engines on Apple Silicon across English and Korean sets, with paired significance tests.
BenchmarkresearchPythonFirst Chinese calibration benchmark for Jev-class models: accuracy, ECE, option order and script robustness.
BenchmarkresearchPythonPre-registered field test of Jev, GLiNER2.5-Decide, Laya and other decision models against small LLMs on fresh September 2026 data.
ProjectresearchPython