jev-pick-and-place-study
MuJoCo pilot where Jev, Claude Haiku and reactive rules choose pick-and-place actions; Jev matched the rules exactly.
Overview
Jev pick-and-place study is a small, reproducible MuJoCo pilot exploring whether Jev can choose actions for a pick-and-place task, compared with Claude Haiku 4.5 and simple reactive rules. Every 0.2 simulated seconds a controller picks one bounded command; shared code handles geometry and executes 5-20 mm movements. Inputs include exact positions, contact and alignment flags and explicit task rules, with no vision or training.
Over ten unseen seeds, each run normally and with a target change after lifting, the README reports rules and Jev both at 10/10 and 10/10, and Haiku at 4/10 and 1/10. Jev's median decision took 146 ms at $0.0035 API cost per success, versus 732 ms and $0.4174 for Haiku. The main finding is that Jev chose exactly the same action sequence as the rules controller in all 20 episodes: it executed the supplied policy reliably, but showed no task advantage from adding AI. The author stresses this is exploratory and not a general model ranking.
What the decision model decides
| Type | Question | Answers |
|---|---|---|
| Choice | Which bounded motion command should the arm execute next?(inferred) | — |
How it works
- Observe positions, contact and alignment flags
- Ask the controller (rules, Jev or Haiku) for one bounded command
- Shared code executes the movement
- Repeat every 0.2 simulated seconds until success or 100 decisions
Strengths
- Honest result that rules sufficed for this task
- Raw traces, protocol and video published
- Spend caps for paid runs
Caveats
- Only 20 episodes per controller
- Engineered interface with explicit rules in the prompt
- Simulation pauses during API calls; not hardware timings
Getting started
uv venv --python 3.11, install requirements, copy .env.example to .env, then python compare.py --method rules --split eval --start 1001 --count 10At a glance
- Type
- Project
- Status
- research
- Language
- Python
- Runtime
- MuJoCo simulation
- Has code
- yes
Use cases and integrations
How curated lists describe it
A small reproducible MuJoCo pilot comparing Jev, Claude Haiku, and reactive rules for pick-and-place.
hellogumbo/awesome-jev
Frequently asked questions
Did Jev beat simple rules?
No. It succeeded as often and chose exactly the same actions as the rules controller.
How did it compare with Claude Haiku?
Jev succeeded in all 20 episodes versus 5 for Haiku in this setup, faster and cheaper.
Can I replay without keys?
Yes. The provided video and traces can be replayed without API calls.
Related projects
MuJoCo Franka arm stacking cubes: code proposes and simulates small moves, Jev picks moves, grasp, release and completion, vs Claude.
AppprototypePythonFixed-wing landing simulator and benchmark where Jev maps four Choice decisions to aircraft controls against humans, baselines and LLMs.
DemoprototypePythonSimulated Franka arm that follows plain-English goals by having Jev chain about 30 hardcoded motion primitives, one per step.
Agent skillprototypePythonCamera-only MuJoCo quadrotor where Jev makes 2.5 Hz tactical judgments while code keeps flight control and a safety veto.
ProjectprototypePythonMulti-drone simulation where Jev makes fast reflex decisions per drone and an optional slower LLM may advise but never takes control.
AppprototypeTypeScriptOpen 4B System I driving model that scores behaviours like cruise, stop, yield and brake from three camera frames each decision.
ProjectresearchPython