jevaluate
Tool where Jev clicks through your web app from plain-language steps, stops below 80% confidence, and a vision model checks pages.
Overview
Jevaluate is a free tool from Maykana that tests a web app the way a person would and stops to ask a human whenever it is unsure. You write the checks as plain sentences such as log in or open the Invoices page. Jev picks the next step from a short list of options, such as click Invoices, and reports its confidence; below 80% it does not click and leaves a note for a person instead. On each page a second, vision-capable model (DeepSeek) inspects one screenshot for problems like text covering a button, unreadably faint text or leftover code on screen, and a short report summarises findings and cost.
On the authors' demo app with three planted problems, it found all three for under one cent. To test the confidence gate, they ran 60 synthetic requests in English, Arabic and mixed Arabic-English: of 10 wrong answers, all were flagged as unsure, and every confident answer was right, though they note 60 is a small test. The repository also includes field notes, gated evals, a browser loop, a UI text judge, a first-click tree test and an agent skill. It requires Node 18+, a TypeSafe key and a DeepSeek key, and is MIT licensed.
What the decision model decides
| Type | Question | Answers |
|---|---|---|
| Choice | Which on-screen action is the next step toward this check (e.g. click Invoices)? | — |
How it works
- Write checks as plain sentences
- Jev picks the next step and reports confidence
- Below 80% confidence, stop and leave a note for a human
- Vision model screens each page; produce a report with cost
Strengths
- Confidence gate stops uncertain actions
- README reports under one cent for the demo run
- All 10 errors in a 60-request test were flagged as unsure
Caveats
- Small validation set
- Needs both TypeSafe and DeepSeek keys
Getting started
git clone https://github.com/ElshinQ/jevaluate && cd jevaluate, install Playwright, export TYPESAFE_API_KEY and DEEPSEEK_API_KEY, then `node scripts/walk.mjs --spec examples/walk-watch.json --watch`.At a glance
- Type
- Agent skill
- Status
- prototype
- Language
- JavaScript
- Runtime
- Node + Playwright
- Has code
- yes
- License
- MIT
Use cases and integrations
How curated lists describe it
Jevaluate: evaluate before you trust. Field notes, runnable scripts and an agent skill for TypeSafe Jev: gated evals, a browser loop, a product walk with DeepSeek vision, a UI text judge and a first-click tree test. Co-authored with Claude Fable 5.1.
logicrw/awesome-jev-projects
Frequently asked questions
When does it stop and ask a human?
Whenever Jev is less than 80% sure of the next step.
What does the vision model check?
Visual problems like overlapping text, faint text or leftover code on screen.
Is the confidence trustworthy?
In the authors' 60-request test all wrong answers were flagged as unsure, but the sample is small.
Related projects
CodexQA browser skill that indexes in-page controls and has Jev pick the operation and target each step, with no screenshots needed.
Agent skillbetaGoal-driven browser agent where each step is one Jev choice over the page's controls, then reviews the run for root causes.
AppbetaTypeScriptChinese UI test framework where Jev scores candidate page elements for each step of a natural-language test case.
ProjectbetaPythonCucumber runner where Jev resolves each Gherkin step against the live page and Playwright executes it, so tests need only the .feature file.
ProjectbetaTypeScriptBrowser tests written as user steps: Jev finds each control once, the path is recorded, and replays run without the model.
ProjectbetaTypeScriptCommunity runner for Midscene Test adding jevAct, jevAssert and jevWaitFor nodes that use Jev over a bounded, non-text action space.
PluginbetaTypeScript