Is Jev as Accurate as Frontier Models at Classification?
OpenRouter benchmark of Jev 1.13 vs Claude Opus 5 on Banking77: 81.0% vs 84.4% accuracy at a fraction of the latency and cost.
This summary is based on how curated lists describe the project; its README was not available to review.
Overview
This OpenRouter blog post runs all 3,080 test utterances of the Banking77 intent-classification dataset through Jev 1.13 and Claude Opus 5. According to the list summary, Jev reached 81.0% accuracy versus 84.4% for Opus 5, with a median latency of 175 ms versus 2,266 ms and a cost of about $0.11 versus $2.42 per 1,000 utterances.
It is a useful reference for teams deciding whether a decision model is good enough for high-volume intent classification: a few points of accuracy traded for an order of magnitude in speed and cost. The article itself was not fetched, so prompt setup, option descriptions and error analysis are not covered here.
What the decision model decides
| Type | Question | Answers |
|---|---|---|
| Choice | Which of the 77 banking intents does this customer utterance express?(inferred) | — |
How it works
- Take the 3,080 Banking77 test utterances
- Classify each with Jev 1.13 and Claude Opus 5
- Compare accuracy, median latency and cost per 1,000
Strengths
- Public dataset makes the comparison reproducible
- Reports accuracy, latency and cost together
Caveats
- Single dataset and task type
- Published by OpenRouter, which serves both models
At a glance
- Type
- Article
- Status
- research
Use cases and integrations
How curated lists describe it
OpenRouter runs all 3,080 Banking77 test utterances through Jev 1.13 and Claude Opus 5: 81.0% vs 84.4% accuracy, 175 ms vs 2,266 ms median, about $0.11 vs $2.42 per 1,000
AnotiaWang/awesome-decision-models
Frequently asked questions
What did the benchmark find?
Per the list summary, 81.0% accuracy for Jev vs 84.4% for Opus 5, at 175 ms vs 2,266 ms median.
How much cheaper was Jev?
About $0.11 versus $2.42 per 1,000 utterances.
Which dataset was used?
Banking77, all 3,080 test utterances.
Related projects
Pre-registered independent evaluation of Jev on intent classification and confidence-gated cascades, with a transparent erratum.
LibraryresearchPythonBenchmark harness pitting Jev against Claude Sonnet 5 and Haiku 4.5 on Banking77 intent sorting with a confidence gate.
ProjectresearchPythonBenchmark of Jev against low-cost LLMs on 130,000 classification decisions across five datasets for accuracy, cost and latency.
ProjectresearchPythonIndependent benchmark of Jev, constrained-output LLMs and baselines on intent and spam tasks: accuracy, latency, cost, calibration.
BenchmarkresearchPythonIndependent benchmark of Jev versus four LLMs on Banking77 routing and prompt injection, plus a gated cascade.
Articlediscussion- jev-arena★ 122
Side-by-side tool comparing Jev and DeepSeek on labelling the same comments for relevance, sentiment and intent, with replay and reports.
ProjectbetaTypeScript