System-One Control Bench
Benchmark where Jev steers an agent through 100 key-and-door grid puzzles, one Choice per move, scored against an exact BFS solver.
Overview
System-One Control Bench asks whether a non-generative decision model can steer an agent across a small grid. The player sees a text grid puzzle and picks the next move from a list of options, reaching a goal, avoiding walls and collecting a key when a locked door blocks the way. A breadth-first solver knows the shortest route from every position, so each move is scored exactly as optimal or not. There are 100 puzzles, ten conditions that change what the player is told, and five rule sets; the puzzles are the final benchmark with no held-out set. A paper, a website with recorded games and a DOI are provided.
The main finding: with full context, Jev picks an optimal move in about 90% of 495 fixed positions, yet finishes only 57 of 100 games. Per the list summary, chat models Gemma 4 26B and DeepSeek V4.1 Flash finished 61 and 60 (80 with reasoning), while open decision models GLiClass and Laya finished 13 and 12. Strong single decisions do not add up to a finished task. Paid players require an explicit --allow-paid flag that states the maximum number of calls, and free baselines (random, greedy, solver) run in seconds. MIT licensed.
What the decision model decides
| Type | Question | Answers |
|---|---|---|
| Choice | Which move is the first step of the shortest path to the target? | north · south · east · west |
How it works
- Render the puzzle state as text with move options
- Ask the player model to choose the next move
- Score the move against the BFS solver
- Play full games and fixed-position exams across conditions
Strengths
- Exact scoring with a solver
- Separates single-decision quality from task completion
- Paid runs guarded by explicit flag
Caveats
- No held-out set
- Grid puzzles may not reflect real tasks
Getting started
uv sync && uv run socb benchmark (free baselines); add TYPESAFE_API_KEY and --allow-paid for Jev.At a glance
- Type
- CLI
- Status
- research
- Language
- Python
- Runtime
- CLI + web viewer
- Has code
- yes
- Has tests
- yes
- License
- MIT
Use cases and integrations
How curated lists describe it
Model evaluation: steers an agent through 100 key-and-door grid puzzles with one Jev `Choice` per move, scoring every pick against an exact BFS solver, and finds that Jev picks an optimal move in 90% of 495 fixed positions yet finishes only 57 of 100 games (Gemma 4 26B 61, DeepSeek V4.1 Flash 60, with reasoning 80; the open decision models GLiClass 13 and Laya 12), so strong single decisions do not add up to a finished task; report and recorded games at [socb.dev](https://socb.dev).
yibie/awesome-jevSystem-One Control Bench: can a non-generative decision model steer an agent across a small grid? 100 puzzles, exact scoring, a leaderboard and a report.
hellogumbo/awesome-jev
Frequently asked questions
How did Jev do?
About 90% optimal single moves but only 57 of 100 puzzles finished.
How did open decision models do?
Per the list summary, GLiClass finished 13 and Laya 12.
What is the lesson?
Strong individual decisions do not guarantee a finished multi-step task.
Related projects
Craftax survival agent where Jev picks one Choice per step from code-listed actions, with optional LLM goals, plus ablations.
ProjectresearchPythonIndependent field guide to Jev 1.13.0 behaviour across 11,621 requests: framing sensitivity, Snake and 3D city driving experiments.
ProjectresearchPython, TypeScriptNine-board gomoku workbench comparing how two Jev players respond to five different input representations of the board.
ProjectresearchTypeScriptRPG experiment pitting Jev against OpenAI's Decisions API as NPCs, comparing wins, per-move accuracy, calibration and cost.
ServiceresearchPythonShort evaluation finding that Jev plays chess poorly, often calling unbalanced positions draws.
ProjectresearchSmall demo collection testing Jev's spatial reasoning with maze single-step choices versus multi-step lookahead.
AppprototypePython