jev-laya-benchmark
Speed and accuracy benchmark of hosted Jev 1.13 versus local Laya 421M on 1,470 synthetic items across eight decision tasks.
Overview
Jev vs Laya benchmark compares Jev (jev-1.13.0, hosted) with Laya (open-weight typed-decision models run locally with MLX on Apple silicon) on 1,470 synthetic items across eight typed-decision tasks, plus controlled latency and throughput sweeps. Data was generated with gpt-5.5 conditioned on labels and blind re-labelled.
The README reports Jev correct on 92.9 percent of 3,386 judgements versus 65.3 percent for Laya 421M, with Choice, Noul and Score accuracies of 99.3, 98.6 and 76.1 for Jev versus 68.3, 71.3 and 51.5. Jev won all 19 questions with paired confidence intervals excluding zero. Laya was about three times faster on a single short question (42 ms versus 136 ms including about 50 ms network), but Jev was faster above three or four questions per request (170 ms for 50 questions versus about 1 second) and on states longer than Laya's 512-token window. A full report covers calibration, long context and multilingual results.
What the decision model decides
| Type | Question | Answers |
|---|---|---|
| Choice | Typed-decision benchmark questions across eight tasks | — |
| Yes / no | Yes/no benchmark questions | — |
| Score | Ordered-scale benchmark questions | — |
How it works
- Generate and blind re-label synthetic data
- Run Jev and Laya with uniform wrappers
- Measure accuracy, calibration and latency sweeps
- Report with bootstrap intervals
Strengths
- Large paired comparison with intervals
- Separates single-question and multi-question latency
- All predictions published
Caveats
- Synthetic data
- Laya tested at 421M size on one Mac
Getting started
See REPORT.md; reproducing needs an Apple silicon Mac, a TypeSafe key and an OpenAI key.At a glance
- Type
- Benchmark
- Status
- research
- Language
- Python
- Runtime
- Benchmark scripts (MLX)
- Has code
- yes
- License
- MIT
Use cases and integrations
How curated lists describe it
Speed and accuracy benchmark: TypeSafe's Jev API vs the local Laya MLX typed-decision model on synthetic tasks
logicrw/awesome-jev-projects
Frequently asked questions
Which is more accurate?
Jev, at 92.9 versus 65.3 percent of judgements correct.
Which is faster?
Laya for a single short question; Jev for many questions or long states.
Is the data real?
No, it is synthetic with blind re-labelling.
Related projects
Apple Silicon workbench running Laya, Decider, Kev and Jev side by side with accuracy, calibration, latency and memory benchmarks.
BenchmarkbetaPythonLocal benchmark comparing Jev with Laya-MLX on classifying GitHub issues for beginner-friendliness, CV fit and difficulty.
BenchmarkresearchPythonBenchmarks Jev against local open decision models on 480 general Persian questions, literary questions and a classical poetry pilot.
BenchmarkresearchPythonTutorial to fine-tune Qwen3.5-4B into a Jev-like classifier with Unsloth on RunPod, serve via vLLM and benchmark against Jev.
BenchmarkresearchPythonDocker recipes and ARM64 images for serving ten native decision checkpoints, such as Cloudflare Clef, behind /v1/systemone on DGX Spark.
ServicebetaPythonToy local Jev-shaped decision API on a 0.5B model that answers yes/no, choice and score questions by scoring option tokens.
ServiceprototypePython