Jev macOS Loop
Native macOS computer-use loop: local OCR and accessibility find controls, Jev picks the next action from a finite list.
Overview
Jev macOS Loop is an open-source computer-use agent for native Mac applications on Apple silicon. Perception runs locally with ScreenCaptureKit, OmniParser CoreML, Apple Vision OCR and macOS accessibility labels, which identify the controls in the selected window. Only observed text, the goal and a finite list of choices are sent to Jev, which selects the next action.
It is aimed at people who want GUI automation without sending screenshots off the machine, and it ships an AGENTS.md so coding agents can install and drive it. Input is guarded by checks on focus, window position, occlusion, frame freshness, confidence and live accessibility state. A separate verifier checks the real result instead of trusting the model's DONE decision. The README reports a Finder demo in which nine files were classified in one request and moved in three group drags in 7.39 seconds. Provider tokens can come from Vercel AI Gateway, OpenRouter or TypeSafe.
What the decision model decides
| Type | Question | Answers |
|---|---|---|
| Choice | Which visible control or action should be taken next toward the goal?(inferred) | — |
| Choice | Which group does each filename belong to?(inferred) | — |
| Yes / no | Is the task done?(inferred) | yes · no |
How it works
- Capture the selected window and identify controls locally with OmniParser, Vision OCR and accessibility
- Send observed text, the goal and finite choices to Jev
- Check focus, occlusion, confidence and accessibility state before acting
- Perform a guarded click or scoped drag
- Verify the outcome independently of Jev's DONE decision
Strengths
- Screenshots, pixels and coordinates stay on the Mac
- Small, finite action choices keep decisions fast
- Guarded input and independent verification reduce wrong actions
- Works with three provider routes
Caveats
- Requires an Apple silicon Mac with macOS 14.2+ and accessibility permissions
- Depends on a hosted Jev provider for decisions
- AGPL-3.0 licence may limit embedding in closed products
Getting started
Paste the provided prompt into a coding agent so it follows the repository's AGENTS.md to install, configure a provider token and run the disposable-app test.At a glance
- Type
- Project
- Status
- beta
- Runtime
- macOS CLI used by coding agents
- Has code
- yes
- Has tests
- yes
- License
- AGPL-3.0
Use cases and integrations
How curated lists describe it
A native macOS automation loop using local OCR and accessibility information with Jev action selection.
logicrw/awesome-jev-projectsNative macOS GUI automation on Apple silicon: OmniParser CoreML and Apple Vision OCR identify controls locally, Jev selects the action. AGPL-3.0.
MrJev/awesome-jev
Frequently asked questions
What does Jev decide in this loop?
It picks the next action from a finite list of controls found locally, and in the Finder demo it classifies filenames into groups.
Are screenshots sent to the cloud?
No. Only observed text, the goal and the finite choices go to the selected provider; pixels stay on the Mac.
Which providers work?
Vercel AI Gateway, OpenRouter or a TypeSafe console token, using your own key.
Related projects
Accessibility-first macOS computer-use MCP server and CLI, with optional Jev guards on targets and inputs before irreversible actions.
PluginbetaPythonLets coding agents run whole goals in your signed-in Chrome on macOS in background tabs, asking Jev or Claude Haiku only for real choices.
ProjectbetaMCP server giving coding agents fast typed decisions (ship gates, risk checks, file triage, page checks) as accept, review or escalate.
MCP serverbetaTypeScriptmacOS computer-use package: a background app driver for any agent, plus a loop where Jev picks each UI action for a bounded subtask.
ProjectbetaPythonWindows MCP server and CLI where Jev observes windows and chooses controls so agents can hand off whole desktop tasks.
MCP serverbetaPythonComputer-use MCP server for macOS and Windows that reads the UI tree and uses Jev to find elements, check screens and wait for states.
MCP serverbetaPython