jev-laya-benchmark
Local benchmark comparing Jev with Laya-MLX on classifying GitHub issues for beginner-friendliness, CV fit and difficulty.
Overview
Jev-laya Benchmark compares TypeSafe's Jev with laya-mlx, an open RLCD-style typed-decision model running locally with MLX, on structured issue classification. A classifier uses Noul scores for beginner-friendliness and fit with a given CV, plus a categorical difficulty choice, checkpointing results atomically.
On 100 issues on a 16 GB Apple Silicon laptop, the README reports Laya slightly faster (mean 309 ms versus 328 ms), while noting Laya was warmed and Jev was not, so latency is directional. Agreement between the two was 80 percent on beginner-friendliness, 42 percent on CV fit and 61 percent on difficulty. Issue data, resumes and outputs are local and not committed. The author describes it as a local experiment, not a production performance claim.
What the decision model decides
| Type | Question | Answers |
|---|---|---|
| Yes / no | Is this issue beginner-friendly? | — |
| Yes / no | Does this issue fit the candidate's CV? | — |
| Choice | How difficult is this issue? | — |
How it works
- Load issues and a resume
- Classify with Jev and with local Laya
- Measure latency and agreement
- Write a report
Strengths
- Direct hosted-versus-local comparison
- Atomic checkpointing
Caveats
- Warm-up mismatch makes latency directional
- Agreement is not accuracy; no gold labels
Getting started
`uv sync`, set TYPESAFE_API_KEY, then `uv run python scripts/benchmark_jev_laya.py --resume ... --limit 100`.At a glance
- Type
- Benchmark
- Status
- research
- Language
- Python
- Runtime
- Scripts with MLX on Apple Silicon
- Has code
- yes
Use cases and integrations
How curated lists describe it
Local benchmark comparing TypeSafe Jev and Laya-MLX for structured issue…
wh000wh000/awesome-jev-live
Frequently asked questions
Which is faster?
Laya was slightly faster in this run, but only Laya was warmed.
Do they agree?
80, 42 and 61 percent on the three questions.
Is this a production claim?
No, the author calls it a local experiment.
Related projects
Apple Silicon workbench running Laya, Decider, Kev and Jev side by side with accuracy, calibration, latency and memory benchmarks.
BenchmarkbetaPythonSpeed and accuracy benchmark of hosted Jev 1.13 versus local Laya 421M on 1,470 synthetic items across eight decision tasks.
BenchmarkresearchPythonBenchmarks Jev against local open decision models on 480 general Persian questions, literary questions and a classical poetry pilot.
BenchmarkresearchPythonTutorial to fine-tune Qwen3.5-4B into a Jev-like classifier with Unsloth on RunPod, serve via vLLM and benchmark against Jev.
BenchmarkresearchPythonDocker recipes and ARM64 images for serving ten native decision checkpoints, such as Cloudflare Clef, behind /v1/systemone on DGX Spark.
ServicebetaPythonToy local Jev-shaped decision API on a 0.5B model that answers yes/no, choice and score questions by scoring option tokens.
ServiceprototypePython