jev-cascade
Research code showing confidence cascades let a 0.8B prefill-only decision model approach a 4B model's accuracy at a fraction of the cost.
Overview
jev-cascade is code for a paper on inference-time compute strategies for prefill-only decision models, Jev-like models that read a full probability distribution over a label menu in one forward pass. The main idea is a confidence cascade: the small model answers over the full menu, and only the least-confident fraction of records is re-answered by a larger model. Because the second pass is just the large model's flat pass, the whole cascade needs two flat passes, with no extra engineering or threshold sweeps.
On HWU-64 with kev-0.8b escalating to kev-4b, the README reports the small model alone at 61.9% accuracy, escalating 25% at 2.0 units of cost reaching 71.6% versus 64.4% for random routing at the same cost, 50% reaching 72.6%, and the large model alone 71.9% at 5.0 units, so escalating a quarter recovers 96% of the large model's accuracy at 40% of the extra cost. Against a 9B model, the same recipe at half the cost lands within one point and beats random routing by 8 points. It also explores shrinking the menu. The paper reports escalation dominating self-re-asking at every cost point across five sources.
What the decision model decides
| Type | Question | Answers |
|---|---|---|
| Choice | Which label from the menu (e.g. HWU-64 intent) applies? | — |
| Other | Is the small model confident enough, or should this record escalate to the larger model? | keep · escalate |
How it works
- Run the small model over all records
- Rank records by confidence
- Re-answer the least-confident fraction with a larger model
- Compare accuracy and cost against random routing
Strengths
- Simple two-pass recipe
- Large accuracy gains for small extra cost
- Shows confidence is a good signal of weakness
Caveats
- Results on open kev models, not hosted Jev
- Benchmarks are intent datasets
At a glance
- Type
- Project
- Status
- research
- Language
- Python
- Runtime
- research scripts
- Has code
- yes
Use cases and integrations
How curated lists describe it
Let a 0.8B Jev-like prefill-only decision model perform like a 4B one: re-run…
wh000wh000/awesome-jev-live
Frequently asked questions
What is a confidence cascade?
Running a small model first and re-answering only its least-confident records with a larger model.
How much does it help?
Escalating 25% recovered 96% of the 4B model's accuracy at 40% of the extra cost, per the README.
Why not re-ask the small model?
The paper finds escalation beats self-re-asking at every cost point.
Related projects
Decision sidecar and Copilot CLI pre-turn hook that mimics Jev choice, noul and score using Azure Foundry Cohere Rerank.
HookprototypePython- cbjev★ 4
Self-hosted encoder model giving Jev-compatible choice, score and noul answers in one pass, pitched as a better-calibrated Laya successor.
ProjectresearchPython Joke decision model trained to answer wrongly with total confidence: 1.9% accuracy at 0.957 mean confidence, via a Jev-style API.
ProjectresearchPythonPre-registered field test of Jev, GLiNER2.5-Decide, Laya and other decision models against small LLMs on fresh September 2026 data.
ProjectresearchPython11 MB CPU-only model that answers choice, noul and score questions with fitted calibration and no LLM at inference.
ProjectresearchPythonTiny pre-registered experiment testing whether a confidence-scoring reward (RLCD-like) yields confidences safe to automate on at 0.9.
ProjectresearchPython