Convex Decision Evals
Convex's eval repo with a 108-question multiple-choice benchmark comparing Jev's Choice answers with 14 LLMs on accuracy, latency and cost.
Overview
Convex Coding Evals is Convex's continuous evaluation suite for how well models write Convex backend code. Alongside the coding evals it contains a decision-model benchmark: 108 verified four-option questions covering 90 of the 112 coding evals, each asked three times with shuffled options. Invalid responses and provider failures count as incorrect.
Jev is queried through OpenRouter's decisions endpoint as a native Choice question, while other models answer A to D through structured output. According to the list summaries, in a September 2026 run jev-1.13 scored 84.6% at a 199 ms median per question and $0.0088 per full run, against 98.0% at 2.12 s and $1.59 for the top model. The README stresses that a correct choice does not show a model can write working Convex code. Results are on a public leaderboard with a per-answer explorer showing Jev probabilities.
What the decision model decides
| Type | Question | Answers |
|---|---|---|
| Choice | Which of the four options correctly answers this Convex question? | A · B · C · D |
How it works
- Validate the decision bank
- Ask each model every question with three option permutations
- Score accuracy per source eval with equal weighting
- Record latency, cost and raw evidence; publish to the leaderboard
Strengths
- Head-to-head accuracy, latency and cost comparison
- Option shuffling controls for position bias
- Raw evidence and per-answer explorer are public
Caveats
- Multiple-choice knowledge does not imply coding ability
- Jev ranked 13th of 15 on accuracy in the cited run
- Unknown provider costs remain unknown
Getting started
With OPENROUTER_API_KEY set: bun run decisions run --provider openrouter --model typesafe/jev-1.13 --limit-evals 2.At a glance
- Type
- Benchmark
- Status
- production
- Language
- TypeScript
- Runtime
- Bun CLI
- Has code
- yes
Use cases and integrations
How curated lists describe it
Leaderboard that asks Jev (native Choice questions through OpenRouter's decisions endpoint) and 14 other models (structured output limited to A-D, no tools, low reasoning effort) 108 verified four-option questions about the Convex backend platform, each asked 3 times per run with shuffled options (random guessing is 25%); in the 23 Sept 2026 benchmark jev-1.13 scored 84.6% ±0.7 (13th of 15) at a 199 ms median per question and $0.0088 per full run, against 98.0% at 2.12 s and $1.59 per run for the top model, GPT-6 Astra. [Leaderboard](https://www.convex.dev/evals/decision), [per-answer explorer](https://convex-evals.netlify.app).
cobanov/awesome-jevLeaderboard that puts Jev and 14 LLMs through 108 four-option multiple-choice questions about the Convex backend platform, comparing accuracy, latency, and cost, with a per-answer explorer showing Jev probabilities.
hellogumbo/awesome-jev
Frequently asked questions
How did Jev score?
The list summary reports 84.6% for jev-1.13 at 199 ms median and $0.0088 per run, versus 98.0% for the top model at much higher latency and cost.
How is Jev queried?
As a native Choice question via OpenRouter's decisions endpoint.
Can I run it locally?
Yes, with an OpenRouter key and the bun run decisions commands; local runs write to output-decisions/.
Related projects
Evaluation suite for decision models with 23,900 pinned rows across nine task families and three primitives, plus a leaderboard.
ProjectbetaPythonOpen-source BYOK arena where Jev faces another model as a judge on your question, with blind voting before costs are revealed.
ProjectbetaTypeScriptpytest-style tool that pins fixtures and expected Jev answers, then fails a model upgrade when answers flip or confidence drops.
ProjectbetaPythonBenchmark harness comparing Jev with OpenRouter models and open Jev-shaped models on accuracy, calibration, latency and cost.
ProjectresearchPythonWrite-up testing whether Jev beats small fast LLM tiers on public and private classification, as a classifier or a filter.
ProjectresearchPythonAudit of 300 web pages for AI citability with nine Jev decisions each, checked against an LLM judge for agreement, cost and latency.
ProjectresearchPython