jev-benchmark
Two Jev benchmarks with a playground: chess (out of lane, about 950 Elo with code-supplied facts) and NPC addressee detection (F1 0.96).
Overview
Jev Benchmark and Playground tests Jev on two deliberately different tasks, with all code, labelled data, raw requests and responses, metrics and an interactive playground, using jev-1.13.0 on 16 September 2026. Chess is chosen as outside Jev's lane because it requires calculation and lookahead; deciding which game NPCs a player is speaking to via speech-to-text is squarely inside it.
On chess, picking from 20-40 legal moves from a raw board was no better than random; with code-supplied facts it beat random by about 65%, and with one-ply tactical facts by about 78%, finding mate-in-one 24% of the time and playing at roughly 950 Elo on a Stockfish-anchored ladder at 0.2 s median per move. On NPC addressee detection, deciding per NPC whether it is spoken to or merely mentioned, it reached F1 0.96 on clean text and 0.93 on lowercase, unpunctuated transcripts with misheard names, precision 1.0 throughout, and 92% exact-set accuracy versus 64% for a fuzzy name-matching heuristic, at 0.17 s median with 8-10 questions per call. The lesson: compute exact answers in code and give Jev the facts. It is MIT licensed.
What the decision model decides
| Type | Question | Answers |
|---|---|---|
| Choice | Which legal chess move should be played? | — |
| Yes / no | Is the player speaking to this NPC (rather than merely mentioning it)? | yes · no |
How it works
- Prepare chess positions and labelled NPC utterances
- Supply state with or without code-computed facts
- Ask Jev typed questions
- Measure accuracy, Elo, latency and tokens
Strengths
- Shows both a failure case and a strong fit
- All raw data published
- Interactive playground
Caveats
- Chess results depend heavily on supplied facts
- Single model version and date
At a glance
- Type
- Benchmark
- Status
- research
- Language
- Python
- Runtime
- benchmark scripts and playground
- Has code
- yes
- License
- MIT
Use cases and integrations
How curated lists describe it
Benchmarks Jev on chess moves and identifying which game NPC a player addresses.
logicrw/awesome-jev-projectsBenchmarks and a playground for TypeSafe.
wh000wh000/awesome-jev-liveBenchmarks and a playground for TypeSafe's Jev (System One) model: chess, and who-is-the-player-talking-to for speech-to-text game NPCs.
hellogumbo/awesome-jev
Frequently asked questions
Can Jev play chess?
Poorly from a raw board; about 950 Elo when code supplies tactical facts.
How well does it detect who a player is talking to?
F1 0.96 on clean text and 0.93 on noisy transcripts, with precision 1.0.
What is the main lesson?
Compute exact answers in code and hand Jev the facts for judgment.
Related projects
One-page web app where Jev plays chess against OpenRouter LLMs, Stockfish or you, showing the moves it weighed and their probabilities.
AppbetaJavaScriptIndependent field guide to Jev 1.13.0 behaviour across 11,621 requests: framing sensitivity, Snake and 3D city driving experiments.
ProjectresearchPython, TypeScriptNine-board gomoku workbench comparing how two Jev players respond to five different input representations of the board.
ProjectresearchTypeScriptRPG experiment pitting Jev against OpenAI's Decisions API as NPCs, comparing wins, per-move accuracy, calibration and cost.
ServiceresearchPythonShort evaluation finding that Jev plays chess poorly, often calling unbalanced positions draws.
ProjectresearchSmall demo collection testing Jev's spatial reasoning with maze single-step choices versus multi-step lookahead.
AppprototypePython