jev-behavior-study
Independent field guide to Jev 1.13.0 behaviour across 11,621 requests: framing sensitivity, Snake and 3D city driving experiments.
Overview
jev-behavior-study is a browsable, independent and AI-assisted field guide to how Jev 1.13.0 behaves, based on live API observations from 16-17 September 2026. It covers 11,621 text-study requests, three Snake game studies, a 3D city driving lab and seven detailed reports, with raw responses and offline verification of report claims.
Its central finding is that performance depends on the exact task and framing: correct answers to prerequisite questions do not always produce correct decisions, and passing a simple task does not establish reliability on a harder one. In the city lab, the corrected pilot completed 0 of 12 full tasks, and under direct control Jev chose straight on all 522 steering calls; the author notes one seed and episode per condition does not establish a general driving score. The Snake studies examine what actually improved with unassisted choices. The README stresses these are synthetic, task-specific results rather than an official benchmark, and that repeated calls are not independent problems. It includes guidance on interacting with Jev and token usage per question, and is MIT licensed.
What the decision model decides
| Type | Question | Answers |
|---|---|---|
| Choice | Which direction should the car steer or the snake move?(inferred) | — |
| Other | How do framing and input conditions change Jev's answers? | — |
How it works
- Design controlled prompt and framing variants
- Run many live requests and record raw responses
- Test in games (Snake, 3D city) under different control schemes
- Publish reports with offline verification
Strengths
- Large number of observations with raw data
- Honest documentation of failures
- Interactive viewer for driving episodes
Caveats
- Synthetic, task-specific results
- Single seeds per driving condition
Getting started
Open the City lab at https://rinnecoder.github.io/jev-behavior-study/city_demo/.At a glance
- Type
- Project
- Status
- research
- Language
- Python, TypeScript
- Runtime
- research scripts and browser viewer
- Has code
- yes
- License
- MIT
Use cases and integrations
How curated lists describe it
An independent Jev 1.13.0 behavior study recording successes and failures across question framing, input conditions and games.
logicrw/awesome-jev-projectsIndependent synthetic-task study of Jev 1.13.0 framing sensitivity and failures, with raw responses and offline report checks.
cobanov/awesome-jevIndependent Jev 1.13.0 behavior study: report, controlled prompt experiments, raw results, and offline verification.
hellogumbo/awesome-jev
Frequently asked questions
What is the main finding?
Jev's performance depends heavily on task and framing; simple-task success does not imply reliability on harder tasks.
Could Jev drive in the 3D city?
Not reliably: 0 of 12 full tasks, and it chose straight on every direct steering call.
Is this an official benchmark?
No, it is an independent, synthetic study.
Related projects
Tic-tac-toe teaching project comparing Jev Choice, Score and Noul designs across three levels of state detail.
ProjectresearchJavaScriptLocal SvelteKit workbench and CLI for organizing, editing, running and comparing Jev experiments defined as JSON.
CLIbetaTypeScriptJev-compatible open System One engine with a Claude Code-style terminal REPL showing live probabilities, plus an experiment harness.
ProjectprototypePythonNine-board gomoku workbench comparing how two Jev players respond to five different input representations of the board.
ProjectresearchTypeScriptIndependent 52-page Chinese report on Jev that checks vendor claims and reproduces 50 Chinese tests, with traceable data.
ProjectresearchPythonRPG experiment pitting Jev against OpenAI's Decisions API as NPCs, comparing wins, per-move accuracy, calibration and cost.
ServiceresearchPython