JevVsLLM
Benchmark of Jev against low-cost LLMs on 130,000 classification decisions across five datasets for accuracy, cost and latency.
Overview
JevVsLLM asks whether general-purpose LLMs can match a decision model like Jev on classification accuracy, latency and cost. Every task is converted into a common multiple-choice format and run through OpenRouter: Jev via OpenRouter's Decisions API, and chat models in two output modes, a single letter or code, or structured JSON with the label name.
The main run covers five datasets with 2,000 examples each. The README reports Jev at 89.0% aggregate accuracy, $0.27 and 288 ms median latency, ahead of DeepSeek Flash (88.0%, $0.55, 395 ms) and others, with no output or API failures. The breakdown is more nuanced: LLMs beat Jev on broad text classification (Banking77, AG News) by several points, while Jev wins on harder multiple-choice or reasoning-style sets (BoolQ, ARC Challenge, MMLU). Letter output was usually as accurate as structured JSON while cheaper, and structured output inflated output tokens 2.4 to 8.4 times. Failures are counted as incorrect. It is useful for anyone choosing between Jev and cheap LLMs.
What the decision model decides
| Type | Question | Answers |
|---|---|---|
| Choice | Which label or answer option is correct for this example? (multiple-choice classification across datasets) | — |
How it works
- Convert each dataset into a common multiple-choice format
- Run Jev through OpenRouter's Decisions API
- Run chat models in letter mode and structured-label mode
- Score accuracy (failures count as wrong), cost and latency
- Break results down by dataset
Strengths
- Large sample (130,000 decisions)
- Compares output formats as well as models
- Reports where Jev loses as well as wins
Caveats
- All calls via OpenRouter; provider rate limits affected some models
- Prompt setups may not be optimal for every LLM
- Five datasets only
At a glance
- Type
- Project
- Status
- research
- Language
- Python
- Runtime
- benchmark scripts
- Has code
- yes
Use cases and integrations
How curated lists describe it
Experiment to compare Jev classification model vs.
wh000wh000/awesome-jev-live
Frequently asked questions
Did Jev win?
It had the best aggregate accuracy, lowest latency and no failures, but LLMs did better on Banking77 and AG News.
Which LLM was closest?
DeepSeek Flash in letter mode, at 88.0% versus Jev's 89.0%.
Is JSON output better for LLMs?
Not here; single-letter output was usually as accurate and cheaper.
Related projects
Pre-registered independent evaluation of Jev on intent classification and confidence-gated cascades, with a transparent erratum.
LibraryresearchPythonBenchmark harness pitting Jev against Claude Sonnet 5 and Haiku 4.5 on Banking77 intent sorting with a confidence gate.
ProjectresearchPythonIndependent benchmark of Jev, constrained-output LLMs and baselines on intent and spam tasks: accuracy, latency, cost, calibration.
BenchmarkresearchPythonOpenRouter benchmark of Jev 1.13 vs Claude Opus 5 on Banking77: 81.0% vs 84.4% accuracy at a fraction of the latency and cost.
ArticleresearchIndependent benchmark of Jev versus four LLMs on Banking77 routing and prompt injection, plus a gated cascade.
Articlediscussion- jev-arena★ 122
Side-by-side tool comparing Jev and DeepSeek on labelling the same comments for relevance, sentiment and intent, with replay and reports.
ProjectbetaTypeScript