jev-docs-benchmarks
Reproducible benchmark comparing Jev with hosted and open decision models on rule application, probabilities, latency and cost.
Overview
This repository measures how reliably decision models apply supplied rules, how their probabilities behave, and what their decisions cost. It compares TypeSafe Jev, Microsoft Decision-1 and OpenAI Decisions with hosted and local open models including Clef and Clef-flash, Decision 3.0, Laya, Kev, Eikos, Jebadiah, Mapika Decider, Simple-Jev and GLiNER, using frozen inputs, inspectable expected answers, raw response archives and reproducible scoring.
The aim is to find where each model works and fails, and whether quality, latency and operating cost justify it for classification, support routing, retrieval and similar tasks. Reports include a 23-configuration ranking, Decision 3.0 flagship and edge comparisons, hosted price comparisons, a Perplexity 27B addition and English and Brazilian Portuguese analyses. The README is explicit that it is not an official vendor benchmark or a production accuracy guarantee, and that schema-valid JSON is not evidence of a correct decision.
What the decision model decides
| Type | Question | Answers |
|---|---|---|
| Other | How accurately, consistently and cheaply does each decision model apply supplied rules?(inferred) | — |
How it works
- Freeze task inputs and expected answers
- Run each hosted or local model and archive raw responses
- Score accuracy and probability behaviour reproducibly
- Compare latency and operating cost across configurations
Strengths
- Broad comparison across commercial and open models
- Raw responses and reproducible scoring
- Covers cost and latency, not just accuracy
Caveats
- Mix of source research and synthetic experiments
- Not a guarantee of production accuracy
Getting started
Start with benchmark/reports/decision3-comparison-v2/analysis.md and the setup guides under benchmark/.At a glance
- Type
- Benchmark
- Status
- research
- Language
- Python
- Runtime
- Benchmark harness
- Has code
- yes
Use cases and integrations
How curated lists describe it
Study documentation and benchmark for various Jev-like products (commercial and open source).
hellogumbo/awesome-jev
Frequently asked questions
Which models are compared?
Jev, Microsoft Decision-1, OpenAI Decisions and many open models such as Clef, Laya, Kev and Decision 3.0.
Is it an official benchmark?
No, it is independent research with stated limitations.
What does it measure?
Rule-application accuracy, probability behaviour, latency and cost.
Related projects
- JevBench★ 268
MIT-licensed benchmark and leaderboard for Jev-class decision models: accuracy, calibration, latency and cost on sealed tiers.
BenchmarkproductionPython Probability-aware evaluation of decision models: calibration, coverage at a fixed error budget and latency, Jev vs GLiNER2.5.
BenchmarkresearchPythonMetamorphic benchmark testing whether decision-model probabilities stay coherent under 50 laws, with no gold labels.
BenchmarkresearchPythonBenchmark of nine local Jev-style decision engines on Apple Silicon across English and Korean sets, with paired significance tests.
BenchmarkresearchPythonFirst Chinese calibration benchmark for Jev-class models: accuracy, ECE, option order and script robustness.
BenchmarkresearchPythonPre-registered field test of Jev, GLiNER2.5-Decide, Laya and other decision models against small LLMs on fresh September 2026 data.
ProjectresearchPython