jev-routing-experiment
Jev as an LLM router on RouterArena and LLMRouterBench: the router beat the best single model, but a no-Jev ablation matched it.
Overview
This repository tests whether Jev can route queries across a pool of LLMs cost-efficiently, using RouterArena's own offline scorer for comparability, plus LLMRouterBench, which provides precomputed scored answers for 13 flagship models so routers are evaluated by lookup with no inference cost.
On eight LLMRouterBench datasets (11,668 queries, evaluated on a held-out 5,835), a router combining kNN retrieval evidence from similar development queries with a Jev difficulty rating reached 62.4% accuracy versus 60.3% for the best single model (GPT-5) at lower cost, or matched GPT-5's accuracy at 33% lower cost. The README is candid that a no-Jev ablation using the evidence alone matched these results, so Jev's difficulty signal did not add measurable value here; the oracle upper bound was 82.6%. A separate zero-cost RouterArena pilot with three cached models is also reported.
What the decision model decides
| Type | Question | Answers |
|---|---|---|
| Score | How difficult is this query (frozen rubric)? | — |
How it works
- Split datasets into dev and held-out eval halves
- Retrieve similar dev queries and read each model's accuracy
- Ask Jev to rate query difficulty
- Select a model trading predicted success against cost
- Score by lookup against precomputed answers
Strengths
- Zero inference cost evaluation
- Directly comparable to public leaderboards
- Includes an honest ablation
Caveats
- Jev's signal added no measurable value over retrieval alone here
- Results depend on precomputed model answers
At a glance
- Type
- Benchmark
- Status
- research
- Language
- Python
- Runtime
- Benchmark scripts
- Has code
- yes
- License
- Apache-2.0
Use cases and integrations
How curated lists describe it
Benchmarking TypeSafe's Jev decision model as a cost-efficient LLM router on RouterArena
logicrw/awesome-jev-projectsBenchmarking TypeSafe's Jev decision model as a cost-efficient LLM router on RouterArena.
hellogumbo/awesome-jev
Frequently asked questions
Did the router beat the best single model?
Yes, 62.4% vs 60.3% at lower cost, but retrieval evidence alone did as well.
So did Jev help?
Not measurably in this setup; the no-Jev ablation matched.
Does evaluation cost money?
No. It uses precomputed answers, so routing is evaluated by lookup.
Related projects
Python tool that measures where to route decisions between Jev and a larger model on your own data, then routes by confidence.
BenchmarkbetaPythonBenchmark of LLM-native structured output (Qwen on Cerebras) versus Jev on seven application tasks: errors, latency, tokens and cost.
BenchmarkresearchDeterministic evidence selector for noisy Codex investigations; its earlier Jev subagent router is kept for the record, not recommended.
ProjectresearchResearch framework (DART) testing whether Jev's probabilities are a reliable signal to answer, escalate or abstain.
ProjectresearchPythonBrowser playground: configure a pool of OpenRouter models, let Jev pick one for a task, then judge whether its pick was right.
AppprototypeJavaScriptClassmethod DevelopersIO independent test of Jev for LLM model routing, reporting 40 successful calls and 10 of 10 correct classifications.
Articleresearch