jev-eval
Benchmark harness comparing Jev with OpenRouter models and open Jev-shaped models on accuracy, calibration, latency and cost.
Overview
jev-eval benchmarks TypeSafe Jev against any OpenRouter model or local checkpoint on labelled classification data, reporting accuracy, calibration (ECE), confidence distribution, latency and cost. It ships five tasks with 300 items each and publishes results for Jev 1.13.0, GPT-5.6 Terra via OpenRouter, and three open models run locally (Laya 421M, open-jev 435M and Kev-0.8B).
The README reports Jev at 0.897 on CLINC150 with 151 options and 0.97 on IMDB polarity, while open-jev beat Jev on Banking77 (0.873 vs 0.780). The author highlights that training on the benchmark decides the winner: models that list Banking77 and SST-5 as training data win there but fall well behind on CLINC150, the same job outside their training lists. Jev's median latency stays roughly flat from 2 to 151 options (0.17 to 0.20 s), and calibration varies by task and model.
What the decision model decides
| Type | Question | Answers |
|---|---|---|
| Choice | Which intent, sentiment or label fits this text? | — |
| Yes / no | Is this review positive? | — |
How it works
- Load labelled task data (300 items each)
- Run Jev, an OpenRouter model or a local checkpoint
- Record predictions, probabilities, latency and cost
- Report accuracy, ECE and latency per task
Strengths
- Compares hosted, LLM and local open models in one harness
- Highlights training-set contamination effects
- Measures calibration, not just accuracy
Caveats
- Five public tasks only
- Local timings on an M-series GPU are not directly comparable to hosted latency
At a glance
- Type
- Project
- Status
- research
- Language
- Python
- Runtime
- Benchmark CLI
- Has code
- yes
Use cases and integrations
How curated lists describe it
Compares Jev and OpenRouter models on labeled tasks for accuracy, calibration, latency and cost.
logicrw/awesome-jev-projectsIndependent Jev versus GPT-5.6 Terra comparison on three labeled classification tasks, reporting accuracy, calibration, latency, and cost.
cobanov/awesome-jev
Frequently asked questions
Which models are compared?
Jev, GPT-5.6 Terra, Laya, open-jev and Kev-0.8B.
What is the main finding?
Open models trained on a benchmark win on it but lose on similar unseen tasks, where Jev stays strong.
Does latency grow with more options?
For Jev, median latency stayed around 0.17 to 0.20 s from 2 to 151 options.
Related projects
Evaluation suite for decision models with 23,900 pinned rows across nine task families and three primitives, plus a leaderboard.
ProjectbetaPythonShort report comparing Jev and many other fast decision classifiers with an LLM on grading section relevance in technical documents.
ProjectresearchLarge rubric-classification benchmark of Jev vs Claude Haiku 4.5, Sonnet 5, OpenJev and many open models, with full price tracking.
BenchmarkresearchPythonOpen-source BYOK arena where Jev faces another model as a judge on your question, with blind voting before costs are revealed.
ProjectbetaTypeScriptpytest-style tool that pins fixtures and expected Jev answers, then fails a model upgrade when answers flip or confidence drops.
ProjectbetaPythonJev-compatible open System One engine with a Claude Code-style terminal REPL showing live probabilities, plus an experiment harness.
ProjectprototypePython