jev-agent-failure-benchmark
Benchmarks Jev against frontier LLMs on Who&When Pro: attributing multi-agent failures to an agent, step and error type.
Overview
jev-agent-failure-benchmark asks whether a fast, cheap decision model can find what broke an AI agent as well as a frontier LLM. It evaluates Jev on the text subset of Who&When Pro, an agent-failure-attribution benchmark: given a failed multi-agent run, predict the responsible agent, the decisive step and the error type. Jev answers three typed Choice questions per trace, scored with the official whowhen_eval scorer at a pinned commit so results are comparable with the paper's LLM baselines.
On all 6,257 text traces the README reports Jev at 73.4 Who, 76.4 When, 23.7 What (error F1) and 31.3 joint, ahead of GPT-5.4, Claude Sonnet 4.6, GLM-5 and Qwen3.5-122B from the paper, for about $1.28 total. It notes that Who and When are constrained choices for Jev while the LLMs free-generate, so What, over the same 17-code taxonomy, is the like-for-like axis, where Jev's 23.7 edges the next best 22.2. A leakage test checks that labels never enter Jev's input. Code is Apache-2.0; the CC-BY-4.0 dataset is downloaded separately.
What the decision model decides
| Type | Question | Answers |
|---|---|---|
| Choice | Which agent is responsible for this failure? | — |
| Choice | Which step was decisive? | — |
| Choice | Which error type (17-code taxonomy) caused it? | — |
How it works
- Download the pinned Who&When Pro text subset
- Ask Jev three Choice questions per trace
- Score with the official evaluator
- Compare with paper baselines and report CIs
Strengths
- Uses the official scorer for comparability
- Large evaluation set (6,257 traces) at about $1.28
- Leakage test guards against label contamination
- Honest note on which axis is like-for-like
Caveats
- Who/When comparisons favour constrained choice
- Baselines are paper-reported, not rerun
Getting started
`uv venv && uv pip install -e ".[dev]"`, add TYPESAFE_API_KEY to .env, download the pinned dataset, then run the benchmark script.At a glance
- Type
- Benchmark
- Status
- research
- Language
- Python
- Runtime
- benchmark scripts
- Has code
- yes
- Has tests
- yes
- License
- Apache-2.0
Use cases and integrations
How curated lists describe it
Who&When Pro (injected agent failures): Jev vs a strong LLM on who / which step / error category.
AnotiaWang/awesome-decision-modelsA benchmark using Jev to attribute multi-Agent failures to an Agent, step and error type.
logicrw/awesome-jev-projectsBenchmarking Jev (Typesafe.ai) against a strong LLM on the Who&When Pro agent-failure-attribution benchmark (text subset).
hellogumbo/awesome-jev
Frequently asked questions
What does the benchmark measure?
How well Jev attributes multi-agent failures to an agent, step and error type.
Did Jev beat frontier LLMs?
The README reports higher scores on all axes, with error type as the fairest comparison (23.7 vs 22.2).
What did it cost?
About $1.28 for all 6,257 traces.
Related projects
Slack-like playground for watching computer-use agents work, with replays, comparable runs and Jev choosing from a menu of visible actions.
ProjectbetaTypeScriptOpen 150M encoder that answers noul, choice and score questions in one pass, trainable on a free Colab T4 in about 30 minutes.
ModelresearchPythonBenchmark asking Jev and a local LLM whether each collusion.wiki page was agent-written; neither beats the majority baseline.
BenchmarkresearchPythonAgent observability platform that finds and fixes agent failures, with an optional Jev preclassifier for conversation checks.
ProjectproductionTypeScript- jev-arena★ 122
Side-by-side tool comparing Jev and DeepSeek on labelling the same comments for relevance, sentiment and intent, with replay and reports.
ProjectbetaTypeScript Evaluation suite for decision models with 23,900 pinned rows across nine task families and three primitives, plus a leaderboard.
ProjectbetaPython