Jev Playground
Benchmarks Jev against Luna, Haiku, Gemini and Astra at choosing legal moves in tic-tac-toe and Connect Four.
Overview
Jev Playground benchmarks TypeSafe's Jev against other models in games with explicit states, legal actions and measurable outcomes. Each game is a state machine whose code owns the rules, legal moves and terminal conditions; the model only chooses among the actions it is given, so every decision is inspectable.
All opponents receive the same state, legal options and tactical instructions through Vercel AI Gateway: Jev via its native evaluation model, GPT-5.6 Luna, Claude Haiku 4.5 and Gemini 3.5 Flash Lite via an evaluation adapter, and GPT-6 Astra via schema-restricted structured output. In tic-tac-toe you play X against five boards at once, each with its own move history and decision log showing option weights and confidence where available. Connect Four is also included. The README notes that option weights are not win probabilities and confidence values are not comparable across providers.
What the decision model decides
| Type | Question | Answers |
|---|---|---|
| Choice | Which legal move should O play? | center · top_left · top_right · bottom_left · bottom_right |
How it works
- The game builds the state and legal moves as Choice options
- Each model receives the same state and instructions
- The validated choice advances the game
- Logs record moves, option weights and confidence
Strengths
- Side-by-side comparison under identical inputs
- Inspectable decisions and logs
- Several games and models
Caveats
- Requires a Vercel AI Gateway key
- Small games; results are not a general ranking
- Confidence is not comparable across providers
At a glance
- Type
- Benchmark
- Status
- prototype
- Language
- TypeScript
- Runtime
- Web app
- Has code
- yes
Use cases and integrations
How curated lists describe it
Model evaluation: benchmarks Jev against Luna, Haiku, and Gemini at choosing validated legal moves in explicit-state games, scoring decision quality and consistency across a sequence of moves.
yibie/awesome-jevBenchmarks Jev against other evaluation models in games with explicit states, legal actions, and measurable outcomes.
hellogumbo/awesome-jev
Frequently asked questions
Which models are compared?
Jev, GPT-5.6 Luna, Claude Haiku 4.5, Gemini 3.5 Flash Lite and GPT-6 Astra.
What does Jev decide?
Which legal move to play from the options given.
Are option weights win probabilities?
No. They are the model's returned Choice distribution.
Related projects
Independent field guide to Jev 1.13.0 behaviour across 11,621 requests: framing sensitivity, Snake and 3D city driving experiments.
ProjectresearchPython, TypeScriptNine-board gomoku workbench comparing how two Jev players respond to five different input representations of the board.
ProjectresearchTypeScriptRPG experiment pitting Jev against OpenAI's Decisions API as NPCs, comparing wins, per-move accuracy, calibration and cost.
ServiceresearchPythonShort evaluation finding that Jev plays chess poorly, often calling unbalanced positions draws.
ProjectresearchSmall demo collection testing Jev's spatial reasoning with maze single-step choices versus multi-step lookahead.
AppprototypePythonProject Zomboid performance patches whose benchmark harness computes numbers in code and, per a list, asks Jev only for the verdict.
BenchmarkbetaJava