jev-plays
Craftax survival agent where Jev picks one Choice per step from code-listed actions, with optional LLM goals, plus ablations.
Overview
jev-plays asks whether a System One model can play a survival and crafting game and how to pair it with an LLM. On Craftax-Classic, deterministic code lists every feasible action with its facts, and Jev picks one Choice per step. Variants compare Jev over 17 primitive actions annotated with one-step facts, Jev over code-generated macro options (mine, craft, fight, flee, drink, sleep, shelter, explore), Jev plus a GPT-5.6-terra planner that writes an objective, standing rules and a short plan, the LLM choosing every step, and random macro actions, over 3 seeds and 2,000 steps.
The lessons are candid: Jev is a good literal matcher, picking the option that matches a fact in the state 92-100% of the time and scoring 14.7 achievements against 5.0 for random. Outside literal matching it has no judgment: it falls back to list position, follows stronger wording rather than weighing options, does not evaluate conditions or count, and followed plan text into danger. Differences of two achievements or fewer are within seed noise. A web UI and per-run logs are included.
What the decision model decides
| Type | Question | Answers |
|---|---|---|
| Choice | Which feasible action or macro option should the agent take this step? | — |
How it works
- Code enumerates feasible actions with facts
- Optionally, an LLM writes an objective, rules and plan
- Jev picks one action per step
- Execute in Craftax and log the episode
- Compare agents across seeds
Strengths
- Clear ablations against random and LLM-only control
- Detailed analysis of how Jev chooses
- Reproducible logs and scripts
Caveats
- Only 3 seeds; small differences are noise
- Far below trained RL agents
- Planner integration can lead Jev into danger
At a glance
- Type
- Project
- Status
- research
- Language
- Python
- Runtime
- Harness and web UI on Craftax
- Has code
- yes
- Has tests
- yes
Use cases and integrations
How curated lists describe it
Gaming: Craftax (Crafter) survival agent where deterministic code lists every feasible action with its facts and Jev picks one `Choice` per step, optionally guided by an LLM-written objective and standing rules; 3-seed ablations compare Jev over macro and raw actions against random, an LLM choosing every step, and Jev plus the planner.
yibie/awesome-jev
Frequently asked questions
What does Jev decide?
One action per step from code-listed feasible actions or macro options.
How well does it play?
About 14.7 achievements versus 5.0 for random on the same options, far below trained RL.
What are its weaknesses?
It matches words literally, cannot count or evaluate conditions, and falls back to list order when nothing matches.
Related projects
Benchmark where Jev steers an agent through 100 key-and-door grid puzzles, one Choice per move, scored against an exact BFS solver.
CLIresearchPythonFifteen live arcade games where Jev and LLMs play each other, led by WikiRace in which Jev picks only real links.
ProjectbetaTypeScriptOne-page web app where Jev plays chess against OpenRouter LLMs, Stockfish or you, showing the moves it weighed and their probabilities.
AppbetaJavaScriptPac-Man where Jev or other System One models choose each move at junctions, with live odds, fallbacks and a leaderboard.
ProjectbetaTypeScriptBrowser chess lab where Jev, LLMs and people play, each Jev turn being one Choice over the legal moves, with Stockfish analysis.
ProjectbetaPythonNiceGUI desktop chess app for evaluating Jev's move choices against Stockfish, random players or itself, with PGN export.
ProjectprototypePython