We Tested Jev on 791 Labeled Decisions Against Four LLMs
Independent benchmark of Jev versus four LLMs on Banking77 routing and prompt injection, plus a gated cascade.
This summary is based on how curated lists describe the project; its README was not available to review.
Overview
This AY Automate blog post reports an independent test of Jev against four LLMs through OpenRouter on 791 labeled decisions: 8-way and 77-way Banking77 intent routing and prompt-injection detection. According to the list summary, Jev matched the small models but trailed GPT-5.6 Terra by about 5 points on 77-way routing.
A cascade using a 0.80 confidence gate, where Jev answers confident cases and escalates the rest to Terra, matched Terra's accuracy at about a quarter of the cost. It is a concrete example of combining a decision model and an LLM.
What the decision model decides
| Type | Question | Answers |
|---|---|---|
| Choice | Which Banking77 intent does this query express (8-way or 77-way)? | — |
| Yes / no | Is this input a prompt injection?(inferred) | yes · no |
How it works
- Run Jev and four LLMs on the same labeled decisions.
- Compare accuracy and cost.
- Apply a 0.80 confidence gate and escalate low-confidence cases to a frontier model.
Strengths
- Independent, labeled comparison.
- Shows a cascade matching frontier accuracy at lower cost.
Caveats
- Known only through a list summary.
- Specific datasets and model versions.
At a glance
- Type
- Article
- Status
- discussion
- Has code
- no
- Has tests
- no
Use cases and integrations
How curated lists describe it
Independent OpenRouter run on 8-way and 77-way Banking77 routing plus prompt-injection detection: Jev matches the small models, trails GPT-5.6 Terra by about 5 points on 77-way routing, and a 0.80 confidence gate that escalates the rest to Terra matches Terra's accuracy at about a quarter of the cost
AnotiaWang/awesome-decision-models
Frequently asked questions
How did Jev compare?
It matched small models but trailed GPT-5.6 Terra by about 5 points on 77-way routing.
What did the cascade achieve?
Terra-level accuracy at about a quarter of the cost with a 0.80 confidence gate.
How many decisions were tested?
791 labeled decisions.
Related projects
Test harness for any Jev-API decision model: accuracy, calibration and robustness probes for reordering, injection and padding.
ServicebetaPythonPre-registered independent evaluation of Jev on intent classification and confidence-gated cascades, with a transparent erratum.
LibraryresearchPythonBenchmark harness pitting Jev against Claude Sonnet 5 and Haiku 4.5 on Banking77 intent sorting with a confidence gate.
ProjectresearchPythonResearch code showing that short, natural-looking context additions can flip decision models like Jev to a chosen wrong answer.
ProjectresearchPythonBenchmark of Jev against low-cost LLMs on 130,000 classification decisions across five datasets for accuracy, cost and latency.
ProjectresearchPythonIndependent benchmark of Jev, constrained-output LLMs and baselines on intent and spam tasks: accuracy, latency, cost, calibration.
BenchmarkresearchPython