jev-calibrate
CLI that checks Jev questions against your labelled examples and grades each: act on it, only sort by it, or rewrite it.
Overview
jev-calibrate tests Jev questions against your own labels, because a question can look right on hand-tried inputs and still fail on labelled data. In the shipped example, a first draft of a frustration question got 18 of 26 labelled messages right, and four of its eight wrong answers came with confidence of 0.94 or more. You write the questions and label examples, and the tool tells you for each question whether its answers can drive a decision on their own (gate), can only be used to sort, or carry no usable signal and should be rewritten.
Reports include accuracy against a target, distribution of examples per class, accuracy within one level and mean error for scores, confidence when right versus wrong, accuracy and coverage at confidence cut-offs such as 0.5, 0.7 and 0.9, stability across repeated runs, and the specific misses. It refuses to grade a question whose classes have too few examples, however good the numbers look. It is an unofficial TypeScript CLI on npm with no runtime dependencies, working with TypeSafe, OpenRouter or any server implementing the same format; it has CI and an MIT licence.
What the decision model decides
| Type | Question | Answers |
|---|---|---|
| Other | Is this Jev question good enough to gate a decision, only to sort, or should it be rewritten? | gate · sort · rewrite |
| Score | Example question under test: how frustrated is the customer? | 0 · 1 · 2 |
How it works
- Write questions and label examples
- Run jev-calibrate check across several runs
- Review accuracy, confidence, coverage at cut-offs and stability
- Act on the verdict: gate, sort or rewrite
Strengths
- Catches confident wrong answers before deployment
- Refuses to grade under-sampled classes
- Works with multiple providers
Caveats
- Requires a labelled set for each question
- Unofficial tool
Getting started
`npm i -g jev-calibrate`, then `jev-calibrate check --runs 3`.At a glance
- Type
- Benchmark
- Status
- beta
- Language
- TypeScript
- Runtime
- Node CLI
- Has code
- yes
- Has tests
- yes
- License
- MIT
Use cases and integrations
How curated lists describe it
Jev is the typed-decision model from TypeSafe: it takes a \`state\` and a set of questions (\`noul\`, \`choice\`, \`score\`) and returns probabilities instead of text. How well a question works depends on its wording, on your data and on the threshold you cut at, and none of the three can be read off a single good-looking answer.
logicrw/awesome-jev-projectsChecks a Jev question against your own labelled examples and grades it: act on it, only sort by it, or rewrite it. Refuses to grade a question whose classes have too few examples, however good the numbers look.
MrJev/awesome-jev
Frequently asked questions
What does jev-calibrate decide?
Whether each of your Jev questions is reliable enough to gate decisions, only to sort, or needs rewriting.
Why not trust a few good answers?
The example shows confident wrong answers that only labelled data reveals.
Which providers work?
TypeSafe, OpenRouter, or any server with the same request format.
Related projects
Claude Code plugin and CLI for the Jev funnel: judge items in bulk, settle confident ones in code, send only uncertain ones onward.
PluginbetaPythonpytest-style tool that pins fixtures and expected Jev answers, then fails a model upgrade when answers flip or confidence drops.
ProjectbetaPythonWrite-up testing whether Jev beats small fast LLM tiers on public and private classification, as a classifier or a filter.
ProjectresearchPythonHarness that runs Jev on all 19,646 TMMLU+ Traditional Chinese questions as four-way Choices, reporting accuracy and calibration.
ProjectresearchPythonReliability testing for typed decision models: contracts, calibration metrics, metamorphic fuzzing and CI pass/warn/fail gates.
DemoprototypePython- reflexbench★ 6
Open benchmark for System One decision models on Choice, Score and Noul, reporting accuracy, calibration and robustness separately.
BenchmarkproductionPython