Decision Index
Reproduction kit for the Decision Index leaderboard: runs 37 public benchmarks against any /v1/systemone engine.
Overview
Decision Index is a benchmark for typed decision engines, models that take a state and choice or noul questions with explicit criteria and return one answer per question with a probability per option. This kit rebuilds the frozen public suite from public sources, runs any engine through it with checkpoint and resume, scores every benchmark with the leaderboard's own scorers and computes the index, on a laptop or as a single Hugging Face Job.
The live board, Decision Index 0.3, ranks by a Full score of 20% public benchmarks, 50% private tests of the same skills and 30% private tasks from new domains, so a public-only advantage counts for little. The public index has 37 benchmarks in five areas, chance-corrected and coverage-adjusted. Models within 0.9 points share a rank, and a separate Vision board ranks image-reading models. Older editions remain reproducible. It is not affiliated with TypeSafe.
What the decision model decides
| Type | Question | Answers |
|---|---|---|
| Choice | Benchmark items posed as choice or noul questions with explicit criteria | — |
How it works
- Rebuild the public suite from sources
- Run an engine via its /v1/systemone endpoint
- Score with the leaderboard's scorers
- Compute the chance-corrected public index; submit for private scoring
Strengths
- Reproducible public part of a leaderboard
- Private tests reduce overfitting to public data
- Works with any compatible endpoint
Caveats
- Most of the Full score comes from private tests you cannot run
- Not affiliated with TypeSafe
Getting started
See the README for running the public suite locally or as a Hugging Face Job against a /v1/systemone endpoint.At a glance
- Type
- Benchmark
- Status
- production
- Language
- Python
- Runtime
- CLI / Hugging Face Job
- Has code
- yes
Use cases and integrations
How curated lists describe it
Reproduction kit for a public leaderboard of typed decision engines: runs the public text suite against any `/v1/systemone` endpoint, locally or as one Hugging Face Job. Not affiliated with TypeSafe.
MrJev/awesome-jev
Frequently asked questions
What does the Decision Index measure?
How well typed decision engines answer choice and noul questions across many benchmarks, chance-corrected.
Can I run it on my model?
Yes, the public part runs against any /v1/systemone endpoint; private parts are run by maintainers.
Is it official?
No, it is independent of TypeSafe.
Related projects
Evaluation suite for decision models with 23,900 pinned rows across nine task families and three primitives, plus a leaderboard.
ProjectbetaPythonJev-compatible open System One engine with a Claude Code-style terminal REPL showing live probabilities, plus an experiment harness.
ProjectprototypePythonLlamaIndex notebook comparing Jev with open decoders, encoders and specialised tools on document-pipeline Choice tasks.
ProjectresearchPythonBenchmark harness comparing Jev with OpenRouter models and open Jev-shaped models on accuracy, calibration, latency and cost.
ProjectresearchPythonReproducible measurements of Jev's real cost, latency, response shapes and accuracy, plus 13 decision models on the same 633 items.
ServiceresearchShort report comparing Jev and many other fast decision classifiers with an LLM on grading section relevance in technical documents.
Projectresearch