Skip to content
Metatext Decide — typed decisions for your agents, with a probability on every answerGet early access
Metatext
Sign Up

WindTunnel

Browser-agent benchmark comparing WebMCP with screenshots, DOM and code execution, including a Jev + Mercury 2.5 configuration.

Benchmarkresearch
View on GitHub ↗

Overview

WindTunnel benchmarks the ways browser agents interact with websites: screenshots (computer use), page structure (DOM and accessibility tree), code execution with Playwright, and WebMCP, where sites expose direct actions. It runs the same tasks on the same sites and measures success, time, tokens and cost. Its headline claim is that WebMCP solves all tasks with 2.5 to 7.5 times faster median execution and 3 to 47 times lower median cost than the median of other methods.

Jev appears as one configuration, paired with Mercury 2.5. On the canonical board of 21 configurations, Jev + Mercury 2.5 with WebMCP solved 49 of 49 tasks and ranked first, while the same pair with ultrafast DOM controls solved 25 of 49 and ranked twelfth. A free harness run with no key just proves it works; the real benchmark needs Docker and a model key.

What the decision model decides

TypeQuestionAnswers
ChoiceWhich page action or WebMCP tool should the agent call next?(inferred)—

How it works

  1. Choose agent configurations and interfaces.
  2. Run the same tasks on the same sites.
  3. Verify outcomes by inspecting the application.
  4. Compare success, time, tokens and cost.

Strengths

  • Controlled comparison across interfaces.
  • Includes a Jev-based configuration.
  • Published run data and methodology.

Caveats

  • Small task set (7 tasks on lightweight sites, 49 attempts per config).
  • Primarily a WebMCP benchmark; Jev is one of many arms.

Getting started

npm ci && WT_FAKE_LIFECYCLE=1 npm run bench

At a glance

Type
Benchmark
Status
research
Runtime
Node + Playwright + Docker
Has code
yes
Has tests
yes

Use cases and integrations

How curated lists describe it

  • Browser-agent benchmark: measures WebMCP against other browser-agent interfaces, with Jev appearing as one of the compared configurations.

    yibie/awesome-jev

Frequently asked questions

How did the Jev configuration do?

49 of 49 with WebMCP (first on the board) and 25 of 49 with DOM controls.

What is WebMCP?

A proposal for websites to expose direct actions to agents.

Can I run it free?

A fake-lifecycle run is free; the real benchmark needs a model key and Docker.

Related projects

  • jev-research-eval

    Reproducible evaluation harness and field note for the Jev Ultrafast research browser, with QC-graded baseline and stress suites.

    ProjectresearchPython
  • playwright-cli-jev-bakeoff

    Reproducible bake-off of Playwright CLI alone, CLI plus Jev, and Playwright MCP on fuzzy browser goals, with honest results.

    MCP serverresearchPython
  • browser-use-olympics

    Browser-agent benchmark with five timed events from one prompt, plus a ~200-line DevTools agent where Jev picks each action.

    ProjectbetaJavaScript
  • ego-jev (jiangkoumo)

    ego-decision-layer: measured, pluggable decision layer for ego lite where Jev picks operation and target in one request.

    ProjectbetaTypeScript
  • jev-browser-bridge

    Plugs any CDP browser, including engines that never draw a page, into a Jev loop that picks operations and targets from the DOM.

    LibrarybetaPython
  • turbo

    Browser automation where Jev picks every step from a sightmap's named actions and reports whether the goal is met.

    ProjectbetaGo