typesafe-ai-benchmark
Benchmark of LLM-native structured output (Qwen on Cerebras) versus Jev on seven application tasks: errors, latency, tokens and cost.
Overview
typesafe-ai-benchmark asks how a fast general LLM compares with a purpose-built judgment model on the same application tasks. It runs Qwen 3.8 27B on Cerebras, using provider structured output with reasoning off, side by side with TypeSafe Jev across seven synthetic workloads, and separately evaluates the local Needle 3 tool-calling model on the same contracts. All outputs are validated before simulated actions.
In the measurements reported for 17 September 2026, Jev had p50/p95 latency of 176/336 ms versus 215/452 ms for Qwen, and estimated API cost of about $0.012 versus $0.31. Fixture agreement was equal on tickets (75/100) and guardrails (100/100), higher for Jev on scoring (100 vs 93) and lower on approvals (95 vs 100) and home control (15 vs 24 of 24). The author notes it is not a controlled speed ranking. Made for Hackers in the Loop, MIT licensed.
What the decision model decides
| Type | Question | Answers |
|---|---|---|
| Choice | Per-workload application decision (ticket routing, guardrail, approval, scoring, home control, driving) | — |
| Yes / no | Native Noul questions mapped to the same application output | yes · no |
How it works
- Compile each workload into a shared output contract.
- Send Qwen one schema-constrained request and Jev one batched Choice/Noul request.
- Validate and map outputs to the same application result.
- Compare agreement, latency, tokens and cost.
Strengths
- Reported Jev cost about 4 percent of Qwen's on the same tasks.
- Lower median and tail latency for Jev in the reported run.
- Raw exports and methodology published.
Caveats
- Synthetic workloads and a single measurement day.
- Accuracy differs by task; Jev was weaker on approvals and home control.
- Not a controlled speed ranking.
At a glance
- Type
- Benchmark
- Status
- research
- Runtime
- browser benchmark theater
- Has code
- yes
- License
- MIT
Use cases and integrations
How curated lists describe it
Compares Jev with other structured-output models on shared application tasks, recording errors, latency, Tokens, and estimated cost.
logicrw/awesome-jev-projectsLLM gateway that mimics the System One output shape for comparison work.
cobanov/awesome-jev
Frequently asked questions
What is compared?
Qwen 3.8 27B structured output on Cerebras versus Jev, plus local Needle 3, on seven application workloads.
What did it find?
Jev was cheaper and faster in the reported run, with mixed accuracy by task.
Why Qwen on Cerebras?
As a low-latency structured-output baseline, not as a ranking of all LLMs.
Related projects
Python tool that measures where to route decisions between Jev and a larger model on your own data, then routes by confidence.
BenchmarkbetaPythonJev as an LLM router on RouterArena and LLMRouterBench: the router beat the best single model, but a no-Jev ablation matched it.
BenchmarkresearchPythonDeterministic evidence selector for noisy Codex investigations; its earlier Jev subagent router is kept for the record, not recommended.
ProjectresearchResearch framework (DART) testing whether Jev's probabilities are a reliable signal to answer, escalate or abstain.
ProjectresearchPythonBrowser playground: configure a pool of OpenRouter models, let Jev pick one for a task, then judge whether its pick was right.
AppprototypeJavaScriptClassmethod DevelopersIO independent test of Jev for LLM model routing, reporting 40 successful calls and 10 of 10 correct classifications.
Articleresearch