MM-JevBench
10,000-sample multimodal benchmark for decision models that return full probability distributions over declared candidates.
Overview
MM-JevBench is a multimodal benchmark for candidate-based decision models. Each sample asks a model to read images, text, audio, video or GUI interaction history and return a probability distribution over a declared candidate set that sums to one. Release MM-JevBench-10k-v1.0 has 10,000 canonical samples (2,000 dev, 8,000 test) in 8,079 leakage-controlled groups from five sources: CMMMU, MULTI, OmniGUI, MMBench-Dev-CN and CCBench.
Every sample is a closed-set choice with 2-7 candidates for visual tasks or 13 fixed action types for GUI next-action decisions; upstream candidates are kept without synthetic distractors and gold labels are one-hot distributions. It evaluates both accuracy and calibration, and the subtitle mentions Jev coherence. It is aimed at teams building multimodal decision models and wanting a common test.
What the decision model decides
| Type | Question | Answers |
|---|---|---|
| Choice | Which candidate answers this multimodal question? | — |
| Choice | GUI next-action type | — |
How it works
- Load dev and test splits.
- Feed each sample's evidence and candidates to the model.
- Collect a probability distribution per sample.
- Score accuracy and calibration by source and domain.
Strengths
- Large, leakage-controlled splits.
- Covers GUI action decisions as well as visual QA.
- Measures calibration, not only accuracy.
Caveats
- Mostly Chinese-language sources.
- All tasks are closed-set choice; no score or yes/no beyond true/false items.
At a glance
- Type
- Benchmark
- Status
- research
- Language
- Python
- Has code
- yes
Use cases and integrations
How curated lists describe it
A multimodal benchmark for evaluating calibrated decision-making, robustness…
wh000wh000/awesome-jev-live
Frequently asked questions
What does MM-JevBench test?
Whether multimodal decision models pick the right candidate and give well-calibrated probabilities.
How big is it?
10,000 canonical samples: 2,000 dev and 8,000 test.
Which sources does it use?
CMMMU, MULTI, OmniGUI, MMBench-Dev-CN and CCBench.
Related projects
Slack-like playground for watching computer-use agents work, with replays, comparable runs and Jev choosing from a menu of visible actions.
ProjectbetaTypeScriptX post by a WebMCP benchmark author reporting Jev plus a small LLM solved 100% of tasks at about 112x lower model cost.
DiscussiondiscussionScreenshot-in, coordinate-out harness where Jev picks among vision-described grid tiles recursively, with probability and margin gates.
ProjectprototypePythonPerplexity's Apache-2.0 multimodal decision model on Qwen3.8-27B, reporting 61.56 on the Decision Index versus 57.9 for Jev.
ServiceproductionPythonSmall open vision-language decision models (2B, 4B, 9B) that answer typed questions about photos and records with a trained can't-tell.
ModelbetaPythonOneJev: open multimodal System One model (0.8B-27B) answering typed questions about screenshots, photos, video and text in one pass.
ModelbetaPython