Jev-ToM
Benchmark release evaluating Theory of Mind in five typed decision models versus seven generative baselines on nine ToM datasets.
Overview
Jev-ToM evaluates whether typed decision models can select among interpretations of a person's knowledge, beliefs, intentions and emotions. It compares five typed models (Jev-1.13, Open-Jev-2B, Valen, Tiny-Jev and OpenJev-0.6B) with seven generative baselines on 11,117 English questions drawn from BigToM, OpenToM, ToMBench, SimpleToM, FANToM, ToMATO, CogToM, Hi-ToM and ToMi, grouped into six capability areas. Invalid responses count as wrong, and the nine benchmarks are weighted equally.
Jev-1.13 has the highest overall accuracy in the cohort at 73.2%, ahead of GPT-3.5-1106 at 71.6% and GPT-5.6-Luna at 70.9%, while the open typed models range from 56.6% down to 43.2%. Further studies examine confidence, supplied reasons (gold-informed distraction interventions), output validity and feedback. The repository provides frozen questions, answer keys, saved predictions and offline scoring tools from an academic group (HKUST KnowComp). The authors note model selection happened after evaluation and the comparison is not exhaustive.
What the decision model decides
| Type | Question | Answers |
|---|---|---|
| Choice | Which candidate interpretation correctly describes this person's belief, knowledge, intention or emotion? | — |
How it works
- Freeze source questions with original candidates and order
- Map questions to six capability groups
- Evaluate with strict validity; invalid counts as wrong
- Aggregate per benchmark with equal weights; run confidence, reason and feedback studies
Strengths
- Large, frozen and reproducible evaluation with saved outcomes
- Compares several open typed models with generative baselines
- Offline scoring, no API needed to reproduce tables
Caveats
- Model selection after evaluation; not exhaustive
- Supplied reasons are machine-reviewed, not human-validated
- Evaluated subsets rather than full benchmark releases
At a glance
- Type
- Project
- Status
- research
- Language
- Python
- Runtime
- Offline scoring tools
- Has code
- yes
Use cases and integrations
How curated lists describe it
Jev-ToM: Evaluating Theory of Mind in Typed Decision Models.
wh000wh000/awesome-jev-live
Frequently asked questions
Which model scored best?
Jev-1.13 at 73.2% overall, slightly ahead of GPT-3.5-1106 and GPT-5.6-Luna in this cohort.
How did open typed models do?
They ranged from 56.6% (Open-Jev-2B) to 43.2% (OpenJev-0.6B).
Can I reproduce the tables without API calls?
Yes, saved predictions and offline scoring tools are included.
Related projects
Evaluation suite for decision models with 23,900 pinned rows across nine task families and three primitives, plus a leaderboard.
ProjectbetaPythonJev-compatible open System One engine with a Claude Code-style terminal REPL showing live probabilities, plus an experiment harness.
ProjectprototypePythonLlamaIndex notebook comparing Jev with open decoders, encoders and specialised tools on document-pipeline Choice tasks.
ProjectresearchPythonBenchmark harness comparing Jev with OpenRouter models and open Jev-shaped models on accuracy, calibration, latency and cost.
ProjectresearchPythonReproducible measurements of Jev's real cost, latency, response shapes and accuracy, plus 13 decision models on the same 633 items.
ServiceresearchShort report comparing Jev and many other fast decision classifiers with an LLM on grading section relevance in technical documents.
Projectresearch