OmniJev
Multimodal robot decision SDK: camera images plus text go to Qwen3.5-9B with a corrective head that picks the next preset arm skill.
Overview
OmniJev is a multimodal research project for robot decision-making. It provides a finite-choice decision SDK, an HTTP service and a MuJoCo robotic-arm workbench with simulation, trajectory replay and reproducible evaluation. Dual-camera images, wrist views and text feed a decision model that scores the candidate options in one forward pass without generating an answer token.
The current model is Qwen3.5-9B with a v4 LoRA and residual corrective head, published on Hugging Face. On 923 held-out image questions the README reports 76.5% total against 72.3% for NeoHorse-Jev-4B and 68.7% for another 4B OmniJev, while noting this is a 9B model against 4B models on a fixed suite, with an untrained letter-logit control not yet run. The list summary adds that released episodes finish transfer, stack and barrier tasks in 13 decisions each and that unobservable inputs return an insufficient-evidence answer. It is for robotics researchers.
What the decision model decides
| Type | Question | Answers |
|---|---|---|
| Choice | Which preset skill should the arm execute next? | — |
| Choice | Which option answers this image question? | — |
How it works
- Capture camera and wrist images plus task text.
- Score preset skill options with the 9B model and corrective head.
- Execute the chosen skill in MuJoCo.
- Replay and evaluate trajectories.
Strengths
- Brings visual state into finite-choice decisions.
- Reproducible simulation and evaluation.
- Honest caveats about comparisons.
Caveats
- Needs the 9B backbone downloaded separately and a capable GPU.
- Preview status; simulation only.
- Comparison is 9B vs 4B on one suite.
Getting started
Place Qwen/Qwen3.5-9B under models/ and the v4 files under models/runs/qwen35-rlcd-v4, then call model.decide(image, question, options, state).At a glance
- Type
- Benchmark
- Status
- research
- Language
- Python
- Runtime
- MuJoCo, HTTP service
- Has code
- yes
Use cases and integrations
How curated lists describe it
Embodied robotics: a Jev-style finite-choice interface that feeds dual-camera images and text to a self-hosted multimodal model and takes the next preset skill for a MuJoCo arm as one typed choice, where released episodes finish transfer, stack and barrier tasks in 13 decisions and 39 output tokens each, 208/208 non-audio probe requests answer correctly, unobservable inputs come back "insufficient evidence" instead of a guess, and four public benchmark pilots hold accuracy equal to a direct short answer while cutting decision latency 6.8–13.6× and total tokens 51–86% (audio input is wired in the client but rejected by the current backend).
yibie/awesome-jev
Frequently asked questions
What does the model decide?
The next preset skill for a simulated robot arm, or the answer to an image question, from supplied options.
How does it compare to NeoHorse-Jev-4B?
On the README's 923-item suite, 76.5% vs 72.3%, though it is a larger model.
Is it Jev?
No, it is a Jev-style interface on an open Qwen backbone.
Related projects
Simulated Franka arm that follows plain-English goals by having Jev chain about 30 hardcoded motion primitives, one per step.
Agent skillprototypePythonCamera-only MuJoCo quadrotor where Jev makes 2.5 Hz tactical judgments while code keeps flight control and a safety veto.
ProjectprototypePythonMulti-drone simulation where Jev makes fast reflex decisions per drone and an optional slower LLM may advise but never takes control.
AppprototypeTypeScriptOpen 4B System I driving model that scores behaviours like cruise, stop, yield and brake from three camera frames each decision.
ProjectresearchPythonExperimental robot loop where Jev chooses the next verified action from perception-generated options for a language goal.
ProjectresearchPythonMuJoCo PiPER arm controlled by a local decider-2b model choosing among ten skills via a Jev-style typed-choice interface while targets move.
Agent skillresearchPython