jev-deterministic-benchmark
1,000-decision benchmark of Jev 1.13 across 25 deterministic reasoning families, showing where it excels and where it fails.
Overview
Jev 1.13 Deterministic Decision Benchmark tests what kinds of decisions Jev handles well. On 22 September 2026 the author ran 1,000 decisions across 25 reasoning families on the JEV-DD-1.0 benchmark, pooling three provider routes (TypeSafe, OpenRouter and another provider that returned only the jev-latest alias). Raw results keep both requested and returned model IDs.
Overall accuracy was 81.7 percent, but the README stresses the pattern behind it. Jev scored around 97 to 100 percent on static rule evaluation such as conditional reasoning, contraposition, quantifiers, transitive comparisons, reachability, source authority, counterexamples and preference ordering. It was much weaker on tasks requiring repeated state updates or exact counting: sequential procedure execution scored 13.2 percent, and most errors there picked a value that had been correct earlier in the sequence. The author notes that jev-latest is a moving alias and later releases need separate evaluation.
What the decision model decides
| Type | Question | Answers |
|---|---|---|
| Choice | Which answer follows deterministically from the given rules or sequence?(inferred) | — |
How it works
- Generate 1,000 deterministic decision items across 25 families
- Query jev-latest through three providers
- Record requested and returned model IDs
- Score accuracy per family and analyse error patterns
Strengths
- Clear map of strong and weak reasoning types
- Provenance of model versions recorded
- Raw CSV published
Caveats
- Single run on one date; alias responses cannot be pinned to a snapshot
- Synthetic deterministic tasks may not reflect real workloads
At a glance
- Type
- Benchmark
- Status
- research
- Language
- Python
- Runtime
- Benchmark scripts
- Has code
- yes
Use cases and integrations
How curated lists describe it
1,000-decision behavioral benchmark of Jev across 25 deterministic reasoning…
wh000wh000/awesome-jev-live1,000-decision behavioral benchmark of Jev across 25 deterministic reasoning families.
hellogumbo/awesome-jev
Frequently asked questions
How accurate was Jev overall?
81.7 percent on 1,000 decisions, per the README.
Where does it struggle?
Tasks that require updating internal state step by step or exact counting, such as sequential procedures at 13.2 percent.
Where is it strong?
Evaluating static situations against clear rules, scoring around 97 to 100 percent.
Related projects
- jev-arena★ 122
Side-by-side tool comparing Jev and DeepSeek on labelling the same comments for relevance, sentiment and intent, with replay and reports.
ProjectbetaTypeScript Evaluation suite for decision models with 23,900 pinned rows across nine task families and three primitives, plus a leaderboard.
ProjectbetaPythonOpen-source BYOK arena where Jev faces another model as a judge on your question, with blind voting before costs are revealed.
ProjectbetaTypeScriptpytest-style tool that pins fixtures and expected Jev answers, then fails a model upgrade when answers flip or confidence drops.
ProjectbetaPythonTest harness for any Jev-API decision model: accuracy, calibration and robustness probes for reordering, injection and padding.
ServicebetaPythonSlack-like playground for watching computer-use agents work, with replays, comparable runs and Jev choosing from a menu of visible actions.
ProjectbetaTypeScript