jev-1.13-mini-benchmark
Small labelled support-triage benchmark of jev-1.13 on OpenRouter covering noul, choice, score, routing, consistency and cost.
Overview
This repository is a small, self-contained benchmark of typesafe/jev-1.13 served through OpenRouter's Decisions API on a labelled support-triage task. Dataset v1.2 has 100 cases (96 English, 4 Turkish) covering all three question types, composite cases, and 68 English agentic-routing cases that add a six-way route choice, an escalation probability and a route-confidence score. Each model gets two full runs to measure consistency, labels are author-assigned, and a clear flag separates obvious cases from judgment calls.
On the core set the README reports noul urgency accuracy of 0.917 at a 0.5 threshold (11/12), choice department 1.000 (13/13), and score frustration 1.000 within one level and 0.846 exact, with zero urgency false positives. Run-to-run consistency showed mean absolute probability change of 0.004 and no decision flips, Turkish cases showed no degradation, cost was about $0.016 per 1,000 decisions, and median latency 0.41 s. The README discusses the single noul miss as a criteria-wording issue and records removed contestable cases.
What the decision model decides
| Type | Question | Answers |
|---|---|---|
| Yes / no | Is this support ticket urgent? | yes · no |
| Choice | Which department should handle this ticket? | — |
| Score | How frustrated is the customer? | — |
| Choice | Which of six routes should this go to? | — |
How it works
- Assemble 100 labelled support cases
- Run jev-1.13 twice via OpenRouter
- Score accuracy per question type and consistency across runs
- Report cost, latency and lessons learned
Strengths
- Measures run-to-run consistency, not just accuracy
- Includes a few non-English cases
- Transparent about label disputes and removed cases
- Very low reported cost per decision
Caveats
- Very small sample sizes per metric (e.g. 12-13 items)
- Author-assigned labels
- Single domain (support triage)
At a glance
- Type
- Benchmark
- Status
- research
- Language
- Python
- Runtime
- benchmark scripts
- Has code
- yes
- Stars
- 1
Use cases and integrations
How curated lists describe it
Mini benchmark of TypeSafe's jev-1.13 structured decision model (OpenRouter Decisions API) on labeled support-triage: noul/choice/score, consistency, cost, lessons learned
wh000wh000/awesome-jev-live
Frequently asked questions
What did the benchmark find?
High accuracy on a small support set (e.g. 13/13 department choices) and almost no run-to-run variation.
How much did it cost?
About $0.016 per 1,000 decisions, per the README.
How reliable are the results?
Indicative only; the per-question samples are small and labels are author-assigned.
Related projects
Third-party check of Jev versus GPT-4o-mini and Claude Sonnet 4.5 routing 60 multilingual photo-shoot booking inquiries.
ProjectresearchTypeScriptC# System One client and benchmark comparing Microsoft-Decision-1, Jev and open-jev on the same labelled support tickets.
LibraryprototypeC#Learning experiment comparing Jev with an OpenAI model on GitHub issue ownership and on AG News, TREC and SNIPS classification.
ModelresearchPythonWeb app that sends identical ticket-routing and alert-triage questions to OpenAI Decisions and Jev side by side.
BenchmarkprototypePythonIndependent test of whether Jev's probabilities stay calibrated on 900 unseen rule-generated tickets plus three public benchmarks.
BenchmarkresearchIndependent benchmark of Jev versus GPT-4.1-mini on 200 South African customer enquiries, showing how question wording changed results.
BenchmarkresearchPython