GUI JEV Harness
Screenshot-in, coordinate-out harness where Jev picks among vision-described grid tiles recursively, with probability and margin gates.
Overview
GUI JEV Harness is an auditable harness that locates a GUI target in a screenshot using Jev as a closed-set decision layer during recursive image search. Because Jev's API defines text and JSON state with no image input, a separately configured vision-capable model describes each grid tile in a goal-blind way, and Jev then chooses among those descriptions. Pixels are never sent to Jev.
At each recursion layer the current crop is divided into a deterministic grid, the vision model describes each tile, and Jev returns a probability distribution over tile IDs plus NO_MATCH. Local code validates the closed candidate set, numeric types, approximate probability sum and model, applies confidence and margin gates, maps the selected tile back to original pixels, and either recurses or refuses. A successful result is a point and bounding box in the input image's coordinates; it never captures the screen or clicks, leaving capture, scaling, freshness and execution to the caller.
What the decision model decides
| Type | Question | Answers |
|---|---|---|
| Choice | Which grid tile contains the target described by the goal (or NO_MATCH)? | — |
How it works
- Split the screenshot crop into a grid
- Vision model describes each tile without seeing the goal
- Jev chooses a tile or NO_MATCH
- Apply confidence and margin gates; recurse into the tile or refuse
Strengths
- Auditable, closed-set decisions
- Refuses rather than guessing
- Never clicks; caller keeps control
Caveats
- Requires a separate vision model via AI Gateway
- Multiple calls per locate
Getting started
python -m pip install -e '.[dev]', export AI_GATEWAY_API_KEY and JEV_GUI_VISION_MODEL, then jev-gui locate screenshot.png --goal "...".At a glance
- Type
- Project
- Status
- prototype
- Language
- Python
- Runtime
- CLI / library
- Has code
- yes
- Has tests
- yes
Use cases and integrations
How curated lists describe it
Computer use: recursive screenshot grounding where Jev returns a `Choice` over grid-tile candidates at each level, and local probability and margin gates decide whether to descend or refuse, emitting only a raster point and bounding box and never clicking.
yibie/awesome-jev
Frequently asked questions
Does Jev see the screenshot?
No, a vision model describes tiles and Jev chooses among descriptions.
Does it click?
No, it returns a point and bounding box only.
What if the target is absent?
Jev can choose NO_MATCH, and gates make the harness refuse.
Related projects
OneJev: open multimodal System One model (0.8B-27B) answering typed questions about screenshots, photos, video and text in one pass.
ModelbetaPythonOpen Jev-style decision models with vision (Qwen3.5 4B/9B LoRA) serving Choice, Score and Noul on /v1/systemone.
LibraryresearchPython10,000-sample multimodal benchmark for decision models that return full probability distributions over declared candidates.
BenchmarkresearchPythonRust MCP server for desktop control via accessibility trees, with an optional decision model (Jev or compatible) for small choices.
MCP serverproductionRustmacOS menu-bar Swift app that turns on-device transcribed speech into Jev typed decisions and macOS actions, confirming hard-to-undo steps.
AppbetaSwiftRust CLI for desktop computer use via accessibility trees, with optional jev-desktop scripts letting Jev pick the control and action.
Agent skillbetaRust