computer_use
Go macOS controller where Jev picks the next accessibility action, the target element and whether the goal is done.
Overview
computer-use-jev drives native macOS applications from Go through the Accessibility API, using a persistent Swift worker. It was extracted from the herbie project's computer_use tool and extended with a TypeSafe decision layer.
Given a natural-language goal, each step gathers real state (apps, windows, last snapshot) and asks Jev one batch of typed questions: which action next, which target token, whether the goal is satisfied, and whether text is needed. Code, not the model, maps the answers to a tool call. Targets are chosen only from tokens present in the current snapshot, and if action confidence drops below a threshold the run stops and reports the distribution instead of acting. Compound goals are first judged by Jev and, if needed, split once by an LLM that has no desktop tools. Tests are hermetic, with fakes for the Swift worker and the TypeSafe API.
What the decision model decides
| Type | Question | Answers |
|---|---|---|
| Choice | Which action should be taken next?(inferred) | apps · windows · activate · snapshot · click · fill · type · press · screenshot |
| Choice | Which target token (from the current snapshot) should the action use? | — |
| Yes / no | Is the goal satisfied? | yes · no |
| Yes / no | Does the goal contain multiple instructions? | yes · no |
How it works
- Collect app list, windows and the latest accessibility snapshot
- Ask Jev the batch of typed questions
- Map answers to a computeruse tool call in code
- Stop and report the distribution if confidence is below threshold; repeat until goal satisfied or max steps
Strengths
- Closed target sets prevent hallucinated elements
- Low-confidence stop instead of blind action
- Dry-run mode and hermetic tests
Caveats
- macOS 14+ only, needs Accessibility and Screen Recording permissions
- Compound goals require an extra LLM provider
- Requires a TypeSafe key
Getting started
go build ./cmd/computer-use-jev, then computer-use-jev -goal 'in TextEdit, type "hello world"' with TYPESAFE_API_KEY set.At a glance
- Type
- Project
- Status
- prototype
- Language
- Go
- Runtime
- CLI on macOS (with Swift worker)
- Has code
- yes
- Has tests
- yes
- License
- MIT
Use cases and integrations
How curated lists describe it
A Go-based macOS controller that asks Jev to choose controls and actions from the accessibility tree.
logicrw/awesome-jev-projectsmacOS computer use driven by Jev (TypeSafe System One) as the decision maker.
wh000wh000/awesome-jev-liveMacOS computer use driven by Jev (TypeSafe System One) as the decision maker.
hellogumbo/awesome-jev
Frequently asked questions
What does Jev decide here?
The next action, which on-screen element to act on, whether the goal is done, and whether text input is needed.
What if Jev is unsure?
The run stops below a confidence threshold and reports the distribution instead of acting.
Can I use it without Jev?
Yes, the computeruse subcommands can be called directly for scripting.
Related projects
Python decision layer for browser and desktop automation: code observes, acts and verifies; the model makes one typed choice per step.
ProjectbetaPythonDurable Android sub-agent: a SQLite-backed worker observes the UI and Jev picks among valid actions, with verification and escalation.
MCP serverbetaPythonWindows UI Automation backend for arc-cua that lets a planner hand off desktop subtasks to a Jev-driven click-and-type loop.
PluginprototypePythonReference framework where Jev picks a concrete computer-use action and its execution route from legal, verifiable candidates.
AppresearchPythonmacOS voice agent where Jev picks each step's operation and target control, escalating to an LLM agent when unsure.
ArticleproductionAutoGPT agent platform, listed as adding Jev decision blocks for typed routing, filtering, scoring and confidence-gated next actions.
ProjectproductionPython