jevbench (Leanmcp)
Open framework and preprint benchmarking typed decision models such as Jev, pplx-decider, Liquid D1 and Clef in six clusters.
Overview
JevBench (Leanmcp) is an open framework for benchmarking typed decision models, accompanied by a preprint, a website and a Hugging Face dataset. It scores models across datasets grouped into six clusters (science, tool calling, business, selection, ranking and judging), normalises each dataset to 0 to 100, averages within clusters, and averages the clusters into a single JevBench Index. Models whose APIs reject some datasets, for example because of too many options or unsupported question types, score zero on those.
In the results table, pplx-decider v1.1 leads with an index of 78.0, followed by Liquid D1 at 77.4 (on five of six clusters), Mercury Decide at 76.6 and Jev 1.13 at 76.5, with Clef, Microsoft Decision 1, GPT-6 Luna Decisions, djev-0.1, Clef-omni, Clef-flash and Kev 4B close behind. Jev scored highest of the listed models on the science cluster (82.5) in that table. Ranking was the weakest cluster across models, mostly in the mid-50s. The listing describes the task set as 12 tasks covering Liquid D1, Jev, djev and Laya among others.
What the decision model decides
| Type | Question | Answers |
|---|---|---|
| Other | Typed decision tasks across science, tool calling, business, selection, ranking and judging | — |
How it works
- Run each model on the benchmark datasets
- Normalise per dataset to 0-100
- Average within six clusters, then across clusters
- Publish the JevBench Index and per-cluster scores
Strengths
- Covers many hosted and open decision models
- Cluster breakdown shows strengths and weaknesses
- Preprint and public dataset
Caveats
- Some models unscored on some clusters
- Unsupported question types count as zero, which penalises narrower APIs
At a glance
- Type
- Benchmark
- Status
- research
- Language
- Python
- Runtime
- Benchmark harness (uv)
- Has code
- yes
Use cases and integrations
How curated lists describe it
Open benchmark for typed decision models: Liquid D1, Jev, djev and Laya on 12 tasks.
hellogumbo/awesome-jev
Frequently asked questions
Which model leads?
pplx-decider v1.1 at 78.0 in the README's table, with Jev 1.13 at 76.5.
Where does Jev do best?
Science, at 82.5 in the table.
Is this the same as Benchmark Heaven's JevBench?
No, it is a separate benchmark from Leanmcp with its own index.
Related projects
Evaluation suite for decision models with 23,900 pinned rows across nine task families and three primitives, plus a leaderboard.
ProjectbetaPythonJev-compatible open System One engine with a Claude Code-style terminal REPL showing live probabilities, plus an experiment harness.
ProjectprototypePythonLlamaIndex notebook comparing Jev with open decoders, encoders and specialised tools on document-pipeline Choice tasks.
ProjectresearchPythonBenchmark harness comparing Jev with OpenRouter models and open Jev-shaped models on accuracy, calibration, latency and cost.
ProjectresearchPythonAgent harness with 100 mocked tools measuring steps to finish tasks when the LLM picks tools versus when Jev picks them.
ProjectresearchTypeScriptExperiments testing Jev ideas against baselines, starting with Jev as a top-10 tool router for MCP agents on three benchmarks.
ProjectresearchPython