jev-benchmark (YidiDev)
Large rubric-classification benchmark of Jev vs Claude Haiku 4.5, Sonnet 5, OpenJev and many open models, with full price tracking.
Overview
This benchmark asks whether Jev genuinely reads and applies multi-clause rubrics, and how it compares on quality and price with Claude Haiku 4.5, Claude Sonnet 5 and OpenJev, a free self-hostable alternative. It spans ten structurally distinct test suites covering rubric-conditioned classification, chained decision execution and exam grading, and fifteen models including NLI and embedding controls and later open-weight additions such as Kev-4B, Nimble-9B, CLM-8B, LAYA, Cygnet, Winnow-Q8, Strands Decider, Clef-27B and Clef-Flash-9B.
The README reports, on suites CT1-8, overall accuracy of 98.84% for Jev, 96.46% for Haiku 4.5, 96.18% for Sonnet 5 and 96.28% for OpenJev, at $0.031, $1.590 and $3.599 per thousand calls for the hosted models and an estimated $0.014 self-hosted for OpenJev. Jev's confidence consistently flagged real errors on 3 of 3 tasks, while both Claude models' confidence was consistently useless. Total metered spend was $110.11 plus small GPU rentals, and every corpus, prediction and dollar is published, including tests that found no difference. It is MIT licensed.
What the decision model decides
| Type | Question | Answers |
|---|---|---|
| Choice | Which rubric category does this item fall into?(inferred) | — |
| Score | What grade does this exam answer earn under the rubric?(inferred) | — |
How it works
- Build ten suites of rubric-based tasks
- Run each model on identical inputs
- Track accuracy, calibration and price
- Add open-weight arms and self-hosted grids
Strengths
- Very broad model coverage
- Full price accounting and raw predictions
- Reports calibration usefulness, not just accuracy
Caveats
- Synthetic rubric suites
- OpenJev cost is an estimate
At a glance
- Type
- Benchmark
- Status
- research
- Language
- Python
- Runtime
- benchmark scripts
- Has code
- yes
- License
- MIT
Use cases and integrations
How curated lists describe it
Rubric-Based Zero-Shot Classification Benchmark: Jev vs Claude Haiku 4.5 vs Claude Sonnet 5 vs OpenJev on rubric-conditioned classification, chained decision execution, and exam grading -- with full price tracking.
logicrw/awesome-jev-projectsRubric-Based Zero-Shot Classification Benchmark: Jev vs Claude Haiku 4.5 vs OpenJev on rubric-conditioned classification, chained decision execution, and exam grading -- with full price tracking.
hellogumbo/awesome-jev
Frequently asked questions
How did Jev do?
98.84% accuracy on CT1-8 versus about 96% for Haiku, Sonnet and OpenJev, per the README.
Was confidence useful?
Jev's confidence flagged real errors on 3 of 3 tasks; the Claude models' did not.
How much did it cost to run?
$110.11 metered API spend plus small GPU rentals.
Related projects
Evaluation suite for decision models with 23,900 pinned rows across nine task families and three primitives, plus a leaderboard.
ProjectbetaPythonBenchmark harness comparing Jev with OpenRouter models and open Jev-shaped models on accuracy, calibration, latency and cost.
ProjectresearchPythonShort report comparing Jev and many other fast decision classifiers with an LLM on grading section relevance in technical documents.
ProjectresearchOpen-source BYOK arena where Jev faces another model as a judge on your question, with blind voting before costs are revealed.
ProjectbetaTypeScriptpytest-style tool that pins fixtures and expected Jev answers, then fails a model upgrade when answers flip or confidence drops.
ProjectbetaPythonJev-compatible open System One engine with a Claude Code-style terminal REPL showing live probabilities, plus an experiment harness.
ProjectprototypePython