Typesafe_chess_eval
Short evaluation finding that Jev plays chess poorly, often calling unbalanced positions draws.
Overview
Typesafe_chess_eval is a brief evaluation of Jev playing chess. The author reports that it does not do well, even though chess is not especially open-ended, and notes the model was probably not optimized for games.
The main observation is that position evaluations often suggest a draw for both sides even in very unbalanced positions, followed by wins, with loss probabilities much lower than expected. The README is short and does not give the method or sample size.
What the decision model decides
| Type | Question | Answers |
|---|---|---|
| Choice | Which move should be played?(inferred) | — |
| Choice | Is this position a win, draw or loss?(inferred) | win · draw · loss |
How it works
- Have Jev play and evaluate chess positions.
- Compare its evaluations with actual outcomes.
- Report observations.
Strengths
- A candid negative result on a game task.
Caveats
- Method and sample size are not documented.
- Informal evaluation.
At a glance
- Type
- Project
- Status
- research
Use cases and integrations
How curated lists describe it
An evaluation of typesafe AI chess. As it turns out, the AI isn't doing really well even though chess is not a particularly open-ended game. Still, it's only a prototype and this probably wasn't optimzied for games.
hellogumbo/awesome-jev
Frequently asked questions
How did Jev do?
Poorly, according to the author.
What was odd about its evaluations?
It often predicted draws in very unbalanced positions.
Is the method documented?
Not in the README.
Related projects
Independent field guide to Jev 1.13.0 behaviour across 11,621 requests: framing sensitivity, Snake and 3D city driving experiments.
ProjectresearchPython, TypeScriptNine-board gomoku workbench comparing how two Jev players respond to five different input representations of the board.
ProjectresearchTypeScriptRPG experiment pitting Jev against OpenAI's Decisions API as NPCs, comparing wins, per-move accuracy, calibration and cost.
ServiceresearchPythonSmall demo collection testing Jev's spatial reasoning with maze single-step choices versus multi-step lookahead.
AppprototypePythonProject Zomboid performance patches whose benchmark harness computes numbers in code and, per a list, asks Jev only for the verdict.
BenchmarkbetaJavaBenchmarks Jev against Luna, Haiku, Gemini and Astra at choosing legal moves in tic-tac-toe and Connect Four.
BenchmarkprototypeTypeScript