jev-measured
Reproducible measurements of Jev's real cost, latency, response shapes and accuracy, plus 13 decision models on the same 633 items.
Overview
jev-measured publishes measured cost, latency and raw output from the live Jev API across twelve realistic use cases, because, as the README says, most circulating numbers are the vendor's own or rate-card arithmetic. Everything is reproducible with your own key, and raw data and scripts are in the repository. Runs are dated September to October 2026 and used typesafe/jev-1.13 via OpenRouter.
Key findings, as reported: a real decision cost $0.000015-0.000030 (about 20 times less than the commonly quoted $0.0004); latency was 458-563 ms from a laptop, about 199 ms of it network floor, and 184 ms median from a US-East server; on 27 labelled tickets Jev tied mistral-small-3.2 at 27/27 rather than winning, but was the only one returning a confidence. Score returns a weighted mean of rubric indices, and Noul has no confidence field. A later section runs thirteen decision models, including d1, Decider, Clef and GPT-6 Luna Decisions, on 633 items.
What the decision model decides
| Type | Question | Answers |
|---|---|---|
| Other | What does a Jev decision actually cost and how long does it take? | — |
| Choice | Support-ticket triage choice compared against chat models on 27 labelled tickets(inferred) | — |
| Other | How accurate are 13 decision models on the same 633 items? | — |
How it works
- Define twelve realistic use cases with fixed prompts
- Call the live Jev API and record cost, tokens, latency and raw output
- Separate network floor from model latency
- Compare accuracy with chat models and other decision models on labelled items
- Publish raw JSON and scripts for re-running
Strengths
- Independent numbers rather than vendor claims
- Raw data and scripts for reproduction
- Practical gotchas such as the OpenRouter decisions endpoint and Score shape
Caveats
- Small labelled set (27 tickets) for the chat-model comparison
- Results reflect specific dates and model versions
- Measurements go through OpenRouter, which adds a gateway hop
Getting started
Bring your own OpenRouter key and re-run the scripts in the repository; raw data is in data/measured.json.At a glance
- Type
- Service
- Status
- research
- Runtime
- Measurement scripts
- Has code
- yes
Use cases and integrations
How curated lists describe it
Measured cost, latency, accuracy and raw output: TypeSafe.
wh000wh000/awesome-jev-liveReproducible measurements of hosted Jev (cost per request across twelve use cases, latency with the network floor separated out, the Score and Noul response shapes, and a 27-ticket labelled comparison with three chat models), plus thirteen decision models on OpenRouter's Decisions API answering the same 633 items; raw answers and scripts are in the repo, and runs dated September to October 2026.
cobanov/awesome-jev
Frequently asked questions
How much does a Jev decision cost?
The README reports $0.000015-0.000030 across twelve use cases, at $0.042 per million input tokens with output not billed.
Is Jev more accurate than chat models?
On 27 labelled tickets it tied, not won, though it alone returned a confidence.
Why does Jev fail on OpenRouter chat completions?
Jev is a decisions model and must be called on the decisions endpoint, not /chat/completions.
Related projects
Evaluation suite for decision models with 23,900 pinned rows across nine task families and three primitives, plus a leaderboard.
ProjectbetaPythonZero-dependency Node proxy that shadows Jev with local decision models and reports whether you can safely swap, per question and policy.
ProxybetaJavaScriptJev-compatible open System One engine with a Claude Code-style terminal REPL showing live probabilities, plus an experiment harness.
ProjectprototypePythonLlamaIndex notebook comparing Jev with open decoders, encoders and specialised tools on document-pipeline Choice tasks.
ProjectresearchPythonBenchmark harness comparing Jev with OpenRouter models and open Jev-shaped models on accuracy, calibration, latency and cost.
ProjectresearchPythonShort report comparing Jev and many other fast decision classifiers with an LLM on grading section relevance in technical documents.
Projectresearch