Jev broke our WebMCP benchmark
X post by a WebMCP benchmark author reporting Jev plus a small LLM solved 100% of tasks at about 112x lower model cost.
This summary is based on how curated lists describe the project; its README was not available to review.
Overview
In this X post, the author of the WebMCP benchmark reports that a combination of Jev and a fast small LLM solved 100% of the benchmark's tasks at roughly 112 times lower model cost than a frontier model using computer use. WebMCP tasks involve agents interacting with websites through MCP-exposed tools.
The result supports the pattern of using a decision model for the many small choices in a web task while a small text model handles text entry. Only the list summary was available, so task count, the specific models, and how cost was computed are not documented here and should be checked in the original post.
What the decision model decides
| Type | Question | Answers |
|---|---|---|
| Choice | Which web action or tool should be taken next?(inferred) | — |
How it works
- Agent works through WebMCP tasks
- Jev makes the step decisions
- A small LLM writes text where needed
- Compare success and cost against a frontier computer-use model
Strengths
- Reported 100% task success at much lower model cost
Caveats
- Social post; methodology not available
- Benchmark author's own report
At a glance
- Type
- Discussion
- Status
- discussion
Use cases and integrations
How curated lists describe it
X: the benchmark's own author reports that Jev plus a fast small LLM solved 100% of WebMCP tasks at roughly 112x lower model cost than a frontier model with computer use.
yibie/awesome-jev
Frequently asked questions
What was reported?
Jev plus a small LLM solved all WebMCP tasks at about 112x lower model cost than a frontier computer-use model.
Who reported it?
The WebMCP benchmark's own author.
Is the setup published?
Not in the list summary; see the post.
Related projects
Slack-like playground for watching computer-use agents work, with replays, comparable runs and Jev choosing from a menu of visible actions.
ProjectbetaTypeScriptZero-dependency Node proxy that shadows Jev with local decision models and reports whether you can safely swap, per question and policy.
ProxybetaJavaScriptReproducible measurements of Jev's real cost, latency, response shapes and accuracy, plus 13 decision models on the same 633 items.
ServiceresearchIndependent measurements of Jev versus open Laya on typed decisions, plus a local-first cascade that matches Jev at 1.8x speed.
ServiceresearchPythonReproducible 2,976-request measurement of whether batching questions in one Jev call bills the state once and keeps answers stable.
BenchmarkresearchPython10,000-sample multimodal benchmark for decision models that return full probability distributions over declared candidates.
BenchmarkresearchPython