jev-baselines-eval
Pre-registered independent evaluation of Jev on intent classification and confidence-gated cascades, with a transparent erratum.
Overview
jev-baselines-eval is an independent, pre-registered evaluation of Jev run on 18 September 2026, two days after launch, by an author with no TypeSafe affiliation who paid about $1 in API costs. The first study tests intent classification on CLINC150 and confidence-gated cascades in which uncertain cases escalate to a bigger model; a second study in the injection folder asks whether the confidence gate survives an authority-framed prompt injection, covering Jev and two open-weight Jev replicas.
Both pre-registered experiments returned ambiguous results under their kill/go criteria, so nothing establishes Jev as a better escalation gate than a small LLM. The data support Jev as a strong zero-shot classifier: +7.5 points over gpt-5.4-nano (paired 95% CI +3.0 to +12.5) and -4.5 points versus GPT-5.6 Terra. Median call duration was about 2.2 times shorter than the nano configuration (0.42 s vs 0.92 s), not the 40-200x vendor comparisons suggest, measured as client-side wall-clock time. A three-round erratum documents corrections found by external review, including a threshold-margin sensitivity that flips the sign of the headline cascade result. It is MIT licensed.
What the decision model decides
| Type | Question | Answers |
|---|---|---|
| Choice | Which of the CLINC150 intents does this utterance express?(inferred) | — |
| Other | Is the classifier confident enough to keep its answer, or should it escalate? | keep · escalate |
How it works
- Pre-register hypotheses and kill/go criteria
- Run Jev and baseline LLMs on the same items
- Build confidence-gated cascades and measure outcomes
- Publish results, external reviews and errata
Strengths
- Pre-registered and independently funded
- Paired confidence intervals
- Unusually transparent error corrections
Caveats
- Both experiments ambiguous under their own criteria
- Small samples (e.g. 30 items for latency)
At a glance
- Type
- Library
- Status
- research
- Language
- Python
- Runtime
- research scripts
- Has code
- yes
- License
- MIT
Use cases and integrations
How curated lists describe it
Every latency number here is \*\*wall-clock duration of one API call\*\* measured in the client: a timestamp before the request, another after the full response body is read (\[\`code/run\_b1.py:32\`\](code/run\_b1.py), \[\`code/common.py:50\`\](code/common.py)). Non-streaming on both sides, so these are completion times, not time-to-first-token.
logicrw/awesome-jev-projects
Frequently asked questions
Is Jev a better escalation gate than a small LLM?
The pre-registered results were ambiguous and do not establish that.
How accurate was Jev at intent classification?
7.5 points above gpt-5.4-nano and 4.5 below GPT-5.6 Terra on CLINC150.
How much faster was it?
About 2.2x shorter median call duration than nano in this setup.
Related projects
Benchmark harness pitting Jev against Claude Sonnet 5 and Haiku 4.5 on Banking77 intent sorting with a confidence gate.
ProjectresearchPythonpytest-style tool that pins fixtures and expected Jev answers, then fails a model upgrade when answers flip or confidence drops.
ProjectbetaPythonTest harness for any Jev-API decision model: accuracy, calibration and robustness probes for reordering, injection and padding.
ServicebetaPythonExperiments testing whether Jev can spot positive-expected-value dice bets from payout tables, and whether recent wins or losses sway it.
ProjectresearchZero-shot run of Jev 1.13 on all seven LexGLUE legal tasks (23,607 examples) versus GPT-5.6 Luna, open models and fine-tuned BERT.
ProjectresearchPythonPre-specified study running Jev and GPT-4.1 as 300 synthetic survey respondents; question format mattered more than model.
ProjectresearchPython