playwright-cli-jev-bakeoff
Reproducible bake-off of Playwright CLI alone, CLI plus Jev, and Playwright MCP on fuzzy browser goals, with honest results.
Overview
playwright-cli-jev-bakeoff is a reproducible harness comparing three agent browser stacks on the same fuzzy goals: Playwright CLI only (reading YAML snapshots and issuing commands), Playwright CLI plus Jev (jev-kit choosing the next action and target ref from slimmed snapshots), and Playwright MCP. Targets are a small local Mini Kanban page and a dense Shopify demo storefront, where the goal is to add a Grey jacket, verify the cart and stop before payment.
On the Shopify target, all arms succeeded, but CLI plus Jev took about 61 s against about 9.8 s for MCP, and Jev looped add_to_cart (13 jackets) without opening the cart until the driver forced it. Estimated cost was about $0.00187 for Jev against $0.00274 for gpt-4o-mini on MCP snapshot tokens, with different pricing bases. The README's honest takeaway is that it did not cleanly reproduce a claim that CLI plus Jev is 98% cheaper and twice as fast as MCP. Research write-ups and artifacts are included, and the Shopify arm never checks out.
What the decision model decides
| Type | Question | Answers |
|---|---|---|
| Choice | Which action and target ref should be used next? | — |
How it works
- Run the same goal with CLI only, CLI plus Jev, and MCP.
- Jev arm picks actions from slimmed snapshots.
- Record success, time, cart contents and cost.
- Write up findings and artifacts.
Strengths
- Side-by-side, reproducible comparison.
- Honest reporting of Jev's failure mode.
- Safe target that stops before payment.
Caveats
- Small number of runs.
- Cost columns use different pricing bases.
Getting started
npm install, cp .env.example .env (set TYPESAFE_API_KEY), npx jev-kit doctor, then serve the kanban page and run the scripts.At a glance
- Type
- MCP server
- Status
- research
- Language
- Python
- Runtime
- Playwright CLI, MCP
- Has code
- yes
Use cases and integrations
How curated lists describe it
Bake-off: Playwright CLI vs CLI+Jev vs Playwright MCP.
wh000wh000/awesome-jev-live
Frequently asked questions
Did CLI plus Jev beat MCP?
Not on the dense Shopify target: MCP was faster and Jev looped adding items.
Which tool drives the Jev arm?
jev-kit, choosing actions from slimmed snapshots.
Does it make purchases?
No, it stops at the cart.
Related projects
Jev-powered browser MCP server that makes each step in one parallel call and escalates to the driving LLM only when unsure.
BenchmarkbetaTypeScriptExperiments testing Jev ideas against baselines, starting with Jev as a top-10 tool router for MCP agents on three benchmarks.
ProjectresearchPythonReproducible evaluation harness and field note for the Jev Ultrafast research browser, with QC-graded baseline and stress suites.
ProjectresearchPythonWeekly go-to-market workflows built on Jev, such as lead triage and deal risk, each tested against Claude.
BenchmarkprototypeBrowser-agent benchmark comparing WebMCP with screenshots, DOM and code execution, including a Jev + Mercury 2.5 configuration.
BenchmarkresearchMCP server and libraries that drive Chrome over CDP and hand a decision model like Jev a bounded menu of page actions.
ProjectproductionTypeScript