local-jev-bench
Benchmark of nine local Jev-style decision engines on Apple Silicon across English and Korean sets, with paired significance tests.
Overview
local-jev-bench runs Jev-style System One decision models locally on an Apple Silicon Mac (an M3 Max with 36 GB) and benchmarks them on the same questions. Engines include Kev, Winnow, Clef, Clef-flash, Von, Jeff, AnyJev, Laya (including a typed-decisions fine-tune), CLM and Decider. Test sets cover English and Korean customer tickets, a 20-way BANKING77 intent subset, Kev's out-of-distribution transfer-v4 set, a typed-decisions leaderboard set, and Korean NSMC movie reviews and KLUE-YNAT headlines.
It reports accuracy, calibration (ECE and KL against gold distributions), option-order stability and latency, and uses paired McNemar tests to say which differences are real. Headline findings: Kev-9B, Kev-4B, Winnow, Decider-4B, Clef and Clef-flash form the top group and are separated only on some sets; Kev's advantage on transfer-v4 comes only from held-out sources resembling its own training data; Laya's typed-decisions fine-tune wins only on its own set; training-free AnyJev matches the top group on some sets but is poorly calibrated on typed-decisions; and Clef-flash leads on BANKING77-20 with the lowest ECE on four of seven sets but is slow on llama.cpp. Documentation is in English and Korean.
What the decision model decides
| Type | Question | Answers |
|---|---|---|
| Choice | Which intent, category or label fits this item? (benchmark tasks) | — |
| Other | Are two engines' accuracy differences statistically significant? (McNemar tests) | — |
How it works
- Install local engines on Apple Silicon
- Run each engine on the same English and Korean sets
- Measure accuracy, ECE, KL, option-order stability and latency
- Compare engines with paired McNemar tests
- Report which differences hold
Strengths
- Many local engines on identical items
- Statistical tests rather than raw rankings
- Korean-language coverage
- Calibration and order-stability measures
Caveats
- Single machine; some latencies from a busier setup
- Some engines' training data is unpublished
- Results may change as models update
At a glance
- Type
- Benchmark
- Status
- research
- Language
- Python
- Runtime
- benchmark scripts on Apple Silicon
- Has code
- yes
Use cases and integrations
How curated lists describe it
Local Jev-style System One decision models on Apple Silicon Macs: AnyJev, Kev…
wh000wh000/awesome-jev-live
Frequently asked questions
Which local engine is best?
Kev, Winnow, Decider, Clef and Clef-flash form a top group that is often statistically tied.
Does it test non-English data?
Yes, Korean tickets, movie reviews and news headlines.
Does it compare with hosted Jev?
The focus is local engines; it notes Cloudflare's own latency figures for Clef.
Related projects
- JevBench★ 268
MIT-licensed benchmark and leaderboard for Jev-class decision models: accuracy, calibration, latency and cost on sealed tiers.
BenchmarkproductionPython Probability-aware evaluation of decision models: calibration, coverage at a fixed error budget and latency, Jev vs GLiNER2.5.
BenchmarkresearchPythonReproducible benchmark comparing Jev with hosted and open decision models on rule application, probabilities, latency and cost.
BenchmarkresearchPythonMetamorphic benchmark testing whether decision-model probabilities stay coherent under 50 laws, with no gold labels.
BenchmarkresearchPythonFirst Chinese calibration benchmark for Jev-class models: accuracy, ECE, option order and script robustness.
BenchmarkresearchPythonPre-registered field test of Jev, GLiNER2.5-Decide, Laya and other decision models against small LLMs on fresh September 2026 data.
ProjectresearchPython