Skip to content
Metatext Decide — typed decisions for your agents, with a probability on every answerGet early access
Metatext
Sign Up

agent-desktop

Rust CLI for desktop computer use via accessibility trees, with optional jev-desktop scripts letting Jev pick the control and action.

Agent skillbetaApache-2.0
View on GitHub ↗

Overview

agent-desktop gives agents reliable desktop control on macOS by reading each app's real UI structure from OS accessibility trees instead of guessing from pixels. It is a single Rust binary with 54 operational commands for observation, interaction, keyboard, mouse, windows and more, a C-ABI library for other languages, compact snapshot references, headless-by-default actions and structured JSON output. Chromium apps can be driven through CDP.

According to the list summaries, since v0.9.2 an optional jev-desktop skill and scripts let Jev choose which control to operate and which action to take from the native accessibility data. The README excerpt here does not detail that part.

What the decision model decides

TypeQuestionAnswers
ChoiceWhich control should be operated next?(inferred)—
ChoiceWhich action should be taken on it?(inferred)—

How it works

  1. Take a skeleton snapshot of the app's accessibility tree
  2. Offer candidate controls and actions to Jev as options
  3. Execute the chosen action by ref
  4. Re-snapshot and repeat

Strengths

  • Accessibility refs are stable and safe to retry
  • Progressive traversal cuts tokens on dense apps (78-96% per the README)
  • Jev is optional

Caveats

  • Jev integration details not in the README excerpt
  • macOS 13+ with Accessibility permission required

Getting started

npm install -g agent-desktop (or npx agent-desktop snapshot --app Finder -i)

At a glance

Type
Agent skill
Status
beta
Language
Rust
Runtime
CLI, C-ABI library, agent skill
Has code
yes
License
Apache-2.0

Use cases and integrations

How curated lists describe it

  • An optional Jev skill for agent-desktop that chooses controls and actions from native accessibility data.

    logicrw/awesome-jev-projects

  • macOS desktop automation over accessibility trees. Since v0.9.2, its jev-desktop scripts let Jev choose which control to operate and which action to take.

    MrJev/awesome-jev

Frequently asked questions

What does Jev decide?

Per the listing, which UI control to operate and which action to take.

Does it work from screenshots?

No, it uses OS accessibility trees; screenshots are optional.

Which OS is supported?

macOS 13 or later.

Related projects

  • zero-use-computer

    Rust MCP server for desktop control via accessibility trees, with an optional decision model (Jev or compatible) for small choices.

    MCP serverproductionRust
  • arc-cua

    macOS computer-use package: a background app driver for any agent, plus a loop where Jev picks each UI action for a bounded subtask.

    ProjectbetaPython
  • Flick

    Local stdio MCP server that executes whole browser and macOS goals with a Jev-driven observe-decide-act loop.

    MCP serverbetaTypeScript
  • jev-adb

    MCP server that lets agents observe and control Android phones over ADB, with Jev picking controls and judging goal completion.

    MCP serverbetaTypeScript
  • jev-browser-skill (ChenYCL)

    Browser and computer use for coding agents via CLI and MCP, with Jev's calibrated judgments and a code-owned control loop.

    MCP serverbetaTypeScript
  • Jev-Desktop

    Windows MCP server and CLI where Jev observes windows and chooses controls so agents can hand off whole desktop tasks.

    MCP serverbetaPython