jevcal
CLI that fits per-question confidence thresholds to a target accuracy on your labelled data, reports escalation share and fails CI on drift.
Overview
jevcal helps answer where to draw the confidence line when a typed decision model like Jev handles confident answers and escalates the rest to a slower LLM. Too high and most traffic escalates, losing the savings; too low and you act on wrong answers. jevcal measures the model on your own labelled data, fits a per-question threshold that meets your accuracy target, verifies it on a held-out split, reports how much traffic each question still escalates and what a cascade would cost versus LLM-only, and fails CI when a model update, for example a moved jev-latest alias, breaks the locked thresholds.
A bundled demo with a built-in simulator shows the output shape: per-question threshold, share handled, accepted accuracy, overall accuracy and ECE, plus which question is the bottleneck for escalation. The README states it includes no Jev benchmark numbers on purpose because TypeSafe's customer agreement restricts publishing performance numbers, which is why the tool lets you measure privately. It also works for anything returning answers with probabilities and can calibrate against an LLM teacher's labels. MIT licensed.
What the decision model decides
| Type | Question | Answers |
|---|---|---|
| Other | Per question, which confidence threshold meets the target accuracy on held-out data? | — |
How it works
- Collect labelled rows (or LLM-teacher labels)
- Run the model and fit per-question thresholds to the target
- Verify on a held-out split; report escalation and cascade cost
- Lock thresholds and fail CI on drift
Strengths
- Thresholds grounded in your own data
- Shows which question bottlenecks escalation
- CI drift protection when model aliases move
- Simulator demo with no key
Caveats
- Needs labelled data or a teacher
- Publishes no Jev numbers by design
Getting started
jevcal demo (built-in simulator, no API key).At a glance
- Type
- Benchmark
- Status
- beta
- Language
- Python
- Runtime
- CLI
- Has code
- yes
- License
- MIT
Use cases and integrations
How curated lists describe it
Model evaluation: fits a per-question confidence threshold to a target accuracy on your own labeled data, verifies it on a held-out split, reports how much traffic still has to escalate to an LLM, and fails CI when a model update breaks the locked thresholds.
yibie/awesome-jevUnofficial CLI that fits a per-question confidence threshold to a target accuracy on your own labeled data, verifies it on a held-out split, shows how much traffic still needs an LLM fallback, and fails CI when a Jev update breaks the locked thresholds
AnotiaWang/awesome-decision-modelsA toolkit for evaluating Jev probabilities on labeled data, selecting confidence thresholds and checking model drift.
logicrw/awesome-jev-projectsFit per-question confidence thresholds on your own labeled data → validate on a…
wh000wh000/awesome-jev-liveFits and drift-checks confidence thresholds against labeled data.
cobanov/awesome-jevStop guessing confidence thresholds: calibrate, threshold, and drift-check typed decision models (TypeSafe Jev) against an LLM teacher.
hellogumbo/awesome-jevPicks the confidence threshold that meets your accuracy target on your own data, and fails CI when a model update breaks it.
MrJev/awesome-jev
Frequently asked questions
What does jevcal output?
A per-question threshold, share handled, accuracy, ECE, escalation rate and cascade cost versus LLM-only.
Why are there no Jev benchmark numbers?
The README says TypeSafe's customer agreement restricts publishing them, so you measure privately.
How does it help with model updates?
It fails CI when a new model version breaks the locked thresholds.
Related projects
Janus (formerly calibre): measures on your data the confidence threshold at which a small model like Jev should defer to a larger one.
ProjectbetaPythonIndependent measurements of Jev versus open Laya on typed decisions, plus a local-first cascade that matches Jev at 1.8x speed.
ServiceresearchPythonPython tool that measures where to route decisions between Jev and a larger model on your own data, then routes by confidence.
BenchmarkbetaPythonLocal-first routing engine for long-running agent tasks that uses a Jev decider to pick a model group per turn.
PluginproductionRust, TypeScriptOpen-source AI gateway for 100+ LLMs that can use Jev to classify requests for complexity routing and to compact tool results.
ProxyproductionPythonOpen-source workspace for running and reviewing AI coding agents across devices, with an optional Jev-based automatic model router.
ProjectproductionTypeScript