jev.sorter
Benchmark harness pitting Jev against Claude Sonnet 5 and Haiku 4.5 on Banking77 intent sorting with a confidence gate.
Overview
jev.sorter is a light Python harness with four stages (decide, gate, check, learn) used to compare TypeSafe's Jev with Claude Sonnet 5 and Claude Haiku 4.5 on Banking77, a dataset of 77 bank customer-message intents. Each model sorts messages into categories defined by written descriptions; a gate checks the category is valid, non-empty, keyword-consistent and above a confidence threshold, and only gated answers are auto-accepted.
The author reports that under a pre-registered rule at the 70% auto-accept level, Jev scored 91.4% accuracy on auto-accepted messages against Sonnet 5's 91.0%, at $0.10 versus $5.25 per 1,000 messages. The learn stage has Sonnet 5 draft improved category descriptions for human approval. A results site shows accuracy, confused pairs, threshold curves, calibration, spend and a cascade. Committed logs let you rerun the analysis for free; total project spend is reported as $16.05.
What the decision model decides
| Type | Question | Answers |
|---|---|---|
| Choice | Which of the 77 Banking77 intents does this customer message belong to?(inferred) | — |
How it works
- Sample tuning (500) and test (1,000) sets from Banking77
- Model picks a category from written descriptions
- Gate checks validity, keywords and confidence threshold before auto-accept
- Analyze accuracy, calibration and cost; refine descriptions in the learn stage
Strengths
- Pre-registered decision rule on a held-out test set
- README reports similar accuracy to Sonnet 5 at about 2% of the cost
- Analysis reruns from committed logs with no API calls
- Gate has its own tests
Caveats
- Single dataset and single author's harness
- Jev called via OpenRouter, not the direct API
Getting started
pip install -r requirements.txt, put OPENROUTER_API_KEY in .env, then `python analyze.py` to rerun the analysis on committed logs.At a glance
- Type
- Project
- Status
- research
- Language
- Python
- Runtime
- CLI
- Has code
- yes
- Has tests
- yes
Use cases and integrations
How curated lists describe it
Building Message Sorter with Jev vs Traditional LLMs.
wh000wh000/awesome-jev-live
Frequently asked questions
What does the benchmark measure?
Accuracy on auto-accepted messages and cost per 1,000 messages for Jev versus Claude models on Banking77.
What was the headline result?
At the 70% auto-accept row, the author reports Jev at 91.4% accuracy versus Sonnet 5 at 91.0%, for $0.10 versus $5.25 per 1,000 messages.
Can I reproduce it without spending money?
Yes, `python analyze.py` reruns the analysis on the committed logs with no API calls.
Related projects
Pre-registered independent evaluation of Jev on intent classification and confidence-gated cascades, with a transparent erratum.
LibraryresearchPythonpytest-style tool that pins fixtures and expected Jev answers, then fails a model upgrade when answers flip or confidence drops.
ProjectbetaPythonTest harness for any Jev-API decision model: accuracy, calibration and robustness probes for reordering, injection and padding.
ServicebetaPythonExperiments testing whether Jev can spot positive-expected-value dice bets from payout tables, and whether recent wins or losses sway it.
ProjectresearchZero-shot run of Jev 1.13 on all seven LexGLUE legal tasks (23,607 examples) versus GPT-5.6 Luna, open models and fine-tuned BERT.
ProjectresearchPythonPre-specified study running Jev and GPT-4.1 as 300 synthetic survey respondents; question format mattered more than model.
ProjectresearchPython