jev-frontier-100
100-question benchmark placing Jev's reasoning at 77% against small Qwen3.5 models under different thinking budgets.
Overview
Jev Frontier 100 is a small benchmark that makes Jev's reasoning limits concrete by comparing it with local Qwen3.5 models under nine settings. It contains 100 original AI-assisted four-choice questions, ten each in customer service, policy rules, discourse, formal logic, relations, mathematics, temporal reasoning, code semantics, algorithms and evidence integration, including 50 counterfactual pairs and 60 executable-oracle answers.
With three trials per question and no majority voting, the README reports Jev 1.13.0 at 77.0 percent, Qwen3.5 2B with a 2,048-token thinking budget at 82.0 percent, and Qwen3.5 4B at 96.7 percent, with bootstrap intervals. Thinking budget matters: 4B with thinking off scores 56.0 percent. Jev does well on short code semantics and formal logic, while the thinking models are stronger on relationship tracking and algorithms. The author stresses the synthetic English task set does not establish a general intelligence ceiling or parameter equivalence.
What the decision model decides
| Type | Question | Answers |
|---|---|---|
| Choice | Which of four options answers this reasoning question? | — |
How it works
- Write 100 four-choice questions across ten domains
- Run Jev and Qwen3.5 at three sizes and three thinking budgets
- Three trials per condition without voting
- Compare with paired bootstrap intervals and per-domain results
Strengths
- Clear comparison against small thinking models
- Counterfactual pairs and executable answers
- Per-domain breakdown
Caveats
- Small synthetic English dataset
- Near-ceiling 4B result shows dataset limits
Getting started
See docs/PROTOCOL.md and results in the repository.At a glance
- Type
- Benchmark
- Status
- research
- Language
- Python
- Runtime
- Benchmark scripts with local models
- Has code
- yes
- License
- MIT
Use cases and integrations
How curated lists describe it
\*\*Jev scores 77.0%; Qwen3.5 2B with a 2,048-token thinking budget scores 82.0%; Qwen3.5 4B with the same budget scores 96.7%.\*\* This small benchmark makes Jev's observed reasoning limits tangible through nine local-model settings.
logicrw/awesome-jev-projects
Frequently asked questions
How did Jev score?
77.0 percent, versus 82.0 and 96.7 percent for Qwen3.5 2B and 4B with a 2,048-token thinking budget.
Where is Jev strongest?
Short code semantics and formal logic, per the README.
Does this prove Jev is weaker than a 4B model?
Only under the stated thinking conditions; with thinking off the 4B model scored 56 percent.
Related projects
Evaluation suite for decision models with 23,900 pinned rows across nine task families and three primitives, plus a leaderboard.
ProjectbetaPythonJev-compatible open System One engine with a Claude Code-style terminal REPL showing live probabilities, plus an experiment harness.
ProjectprototypePythonLlamaIndex notebook comparing Jev with open decoders, encoders and specialised tools on document-pipeline Choice tasks.
ProjectresearchPythonBenchmark harness comparing Jev with OpenRouter models and open Jev-shaped models on accuracy, calibration, latency and cost.
ProjectresearchPythonReproducible measurements of Jev's real cost, latency, response shapes and accuracy, plus 13 decision models on the same 633 items.
ServiceresearchShort report comparing Jev and many other fast decision classifiers with an LLM on grading section relevance in technical documents.
Projectresearch