jev-bench
Standard-library Python workbench for repeatable classifier latency and accuracy experiments across Jev, Laya and classifier.dev models.
Overview
Jev Bench is a classifier workbench for repeatable API experiments: you configure a dataset and one or more named models, and it measures request latency and label correctness and can publish selected results to a results site. It runs on Python 3.10+ with only the standard library. Adapters include OpenRouter's decisions API (for example typesafe/jev-1.13) and classifier.dev, whose anonymous models need no key.
Experiment files set rounds, warmups, seed, per-request timeouts and a maximum total time; dry runs validate configuration and print the request budget without network calls, and each run writes to a fresh results directory that is never overwritten. Published results include Jev classification accuracy on BANKING77, an OpenRouter comparison and a Jev versus Laya comparison. The README warns that live OpenRouter requests can incur charges and send dataset text to the provider. Code is Apache-2.0, with attribution notices for an MIT-licensed protocol validation and the CC BY 4.0 BANKING77 dataset.
What the decision model decides
| Type | Question | Answers |
|---|---|---|
| Choice | Which label applies to this text (e.g. BANKING77 intent or feedback category)?(inferred) | — |
How it works
- Write an experiment config selecting a dataset and models
- Dry-run to check the request budget
- Run live with warmups and rounds
- Review latency and accuracy; publish selected results
Strengths
- No dependencies beyond the standard library
- Reproducible configs with seeds and budgets
- Some comparisons need no API key
Caveats
- Live runs may cost money and send data to providers
- Results are specific to configured datasets
Getting started
`python3 -m jev_bench --dry-run`, then `python3 -m jev_bench --config experiments/jev-vs-classifier.json` with OPENROUTER_API_KEY set.At a glance
- Type
- Benchmark
- Status
- beta
- Language
- Python
- Runtime
- CLI
- Has code
- yes
- License
- Apache-2.0
Use cases and integrations
How curated lists describe it
A configurable classifier workbench for repeatable latency and accuracy experiments.
hellogumbo/awesome-jev
Frequently asked questions
What does Jev Bench measure?
Request latency and label correctness for configured models on a dataset.
Can I run it without keys?
Yes for classifier.dev models and dry runs; Jev via OpenRouter needs a key.
Where are results?
On jev-bench.cocode.dk and in the results/published folder.
Related projects
- jev-arena★ 122
Side-by-side tool comparing Jev and DeepSeek on labelling the same comments for relevance, sentiment and intent, with replay and reports.
ProjectbetaTypeScript Evaluation suite for decision models with 23,900 pinned rows across nine task families and three primitives, plus a leaderboard.
ProjectbetaPythonOpen-source BYOK arena where Jev faces another model as a judge on your question, with blind voting before costs are revealed.
ProjectbetaTypeScriptpytest-style tool that pins fixtures and expected Jev answers, then fails a model upgrade when answers flip or confidence drops.
ProjectbetaPythonTest harness for any Jev-API decision model: accuracy, calibration and robustness probes for reordering, injection and padding.
ServicebetaPythonSlack-like playground for watching computer-use agents work, with replays, comparable runs and Jev choosing from a menu of visible actions.
ProjectbetaTypeScript