DecisionBench
Evaluation suite for decision models with 23,900 pinned rows across nine task families and three primitives, plus a leaderboard.
Overview
DecisionBench is an evaluation ecosystem for decision models. It provides a frozen suite of 23,900 rows across nine task families and the three primitives (Noul, Choice, Score), a reasoning track, supported runners for models with native decision-token readouts, and a Hugging Face leaderboard where reviewed results can be filtered by task, family, domain or primitive.
According to the list summary, it scores answers on pinned document-grounded tasks and counts malformed probability distributions as misses, so comparisons between models stay reproducible. The example smoke test runs Bosun v3.1 0.6B through the run-hf runner. Documentation covers metrics, artifacts, comparability rules and how to add adapters, tasks and result records. Its documentation covers supported adapters, task views, loading results, metrics and comparability rules, and how to contribute new models, tasks and result records.
What the decision model decides
| Type | Question | Answers |
|---|---|---|
| Yes / no | Document-grounded yes/no questions(inferred) | yes · no |
| Choice | Document-grounded multiple-choice questions(inferred) | — |
| Score | Document-grounded ordinal rating questions(inferred) | — |
How it works
- Install DecisionBench
- Download a supported model at a pinned revision
- Run the frozen task spec with the model's runner
- Inspect and submit results to the leaderboard
Strengths
- Large pinned suite covering all three primitives
- Malformed outputs counted as misses for fair comparison
- Public leaderboard with filtering
Caveats
- Only models with supported native readouts run directly
- Requires model downloads
Getting started
pip install git+https://github.com/Hanno-Labs/decision-bench.git, then decision-bench run-hf task_specs/decisionbench-dev.toml <model> <out> --smoke.At a glance
- Type
- Project
- Status
- beta
- Language
- Python
- Runtime
- CLI
- Has code
- yes
Use cases and integrations
How curated lists describe it
Model evaluation: scores Jev Noul, Choice, and Score answers on pinned document-grounded tasks, counting malformed probability distributions as misses so model comparisons remain reproducible.
yibie/awesome-jev
Frequently asked questions
How big is the suite?
23,900 rows across nine families and three primitives.
How are malformed answers handled?
They count as misses, per the list summary.
Is there a leaderboard?
Yes, on Hugging Face Spaces.
Related projects
Benchmark harness comparing Jev with OpenRouter models and open Jev-shaped models on accuracy, calibration, latency and cost.
ProjectresearchPythonShort report comparing Jev and many other fast decision classifiers with an LLM on grading section relevance in technical documents.
ProjectresearchLarge rubric-classification benchmark of Jev vs Claude Haiku 4.5, Sonnet 5, OpenJev and many open models, with full price tracking.
BenchmarkresearchPythonOpen-source BYOK arena where Jev faces another model as a judge on your question, with blind voting before costs are revealed.
ProjectbetaTypeScriptpytest-style tool that pins fixtures and expected Jev answers, then fails a model upgrade when answers flip or confidence drops.
ProjectbetaPythonJev-compatible open System One engine with a Claude Code-style terminal REPL showing live probabilities, plus an experiment harness.
ProjectprototypePython