What is agent-desktop?
agent-desktop lets an AI agent operate desktop apps without screenshots or pixel matching. It reads an app's accessibility tree, the labelled list of buttons, fields and menus that screen readers use, and returns it as JSON with a reference for every element the agent can act on. The agent clicks or types on a reference, reads the UI again to check, and repeats. There is no model inside; the loop lives in your agent.
It is a Rust binary with 61 command names (57 operational) covering snapshots, ref actions, keyboard and mouse, windows, notifications, the clipboard and sessions, plus a C library for in-process use. Since v1.2.0 on 5 November 2026, agent-desktop mcp serves every command as an MCP tool over stdio. A sandbox command manages local macOS virtual machines so the agent can work on a desktop that is not yours. It is the desktop sibling of Vercel's agent-browser.
Vercel Labs publishes it under Apache-2.0. The repository was created in February 2026, has about 22,000 stars and 30 releases, and the npm package had about 50,600 downloads in the week to 10 November.
How to install agent-desktop
npm install -g agent-desktop && agent-desktop mcpIs agent-desktop worth it?
agent-desktop gets the main design choice right and then documents its edges in unusual detail. References are tied to the snapshot that produced them, so a bare @e8 is rejected and a changed UI returns STALE_REF or AMBIGUOUS_TARGET instead of a click on the wrong control. Errors carry a retry disposition, and the docs tell agents to retry only when it says safe. For dense apps such as Slack, a shallow skeleton snapshot followed by a drill-down into one region keeps the token cost of each read down.
Against Munim Computer Use, the closest rival, the trade is breadth for browser reach. agent-desktop has more commands, a C library, a CLI that agents can call without MCP, and about 22,000 stars against Munim's 37 or so, but it deliberately stops at desktop apps: web pages belong to agent-browser, which it connects to by opening a DevTools port on Chromium and Electron apps you launch. Munim drives your signed-in Chrome directly and has a per-app policy file; agent-desktop has no equivalent allow and block list in its README.
Headless behaviour differs by platform, and each platform page says exactly how. Ref actions use the accessibility API without moving the pointer by default, and --headed allows physical input. On Windows, headless type and right-click are refused, and an app running as administrator can be read but not driven from a normal terminal. On Linux it needs Ubuntu 24.04 or later with a running AT-SPI bus. The Windows binary is unsigned, so SmartScreen and Smart App Control may block it.
The security position is plain in the README: snapshots, clipboard reads and screenshots can contain private desktop content and should stay out of public logs. Anything the agent reads on screen, an email or a chat message, can carry a prompt injection with your whole desktop behind it. The macOS sandbox, which runs the agent in a local VM, is the real mitigation, though the docs say it separates the desktop, not the network or the host, and Linux and Windows have no local sandbox provider. A project that grew this fast with 220 open issues and pull requests will keep changing; the MCP server is less than a week old.
Where it earns its place
An operations analyst at a 30-person logistics firm with a Windows-only freight app
Copy shipment details from emails into the freight app every morning
The agent snapshots the app's form, fills fields with set-value, which works headless on Windows, and re-reads the form to confirm. Free apart from model tokens; run it on a dedicated Windows machine because the agent can reach every open app.
A QA engineer on a macOS desktop app team
Have a coding agent check a new settings screen after each change
Add agent-desktop mcp to Claude Code, then ask the agent to launch the build, open settings and verify each toggle with wait --predicate. Using the macOS VM sandbox keeps test runs off the engineer's own desktop.
A developer building an Electron app with web content
Test both the native menus and the web view in one agent loop
agent-desktop drives the menus and windows, and opens a DevTools port that agent-browser attaches to for the page content. Both are free; expect two tools in the agent's context.
Before you commit
- Setup
- npm install -g agent-desktop (or brew install agent-desktop on macOS and Linux), then agent-desktop status and, on macOS, agent-desktop permissions --guide for the Accessibility grant. For MCP clients, add the command agent-desktop with the argument mcp; skills install with npx skills add.
- Integrations
- Every command is an MCP tool (desktop_snapshot, desktop_click, desktop_session_start and so on), most of them write actions: click, type, set-value, drag, launch, close-app, clipboard-set, notification actions. Works with any agent that can run a CLI; a C library covers in-process use.
- Data and privacy
- Runs locally; the MCP server speaks stdio and opens no port, though it opens a DevTools port on Chromium and Electron apps you launch for agent-browser. Snapshots, clipboard and screenshots can hold private content and go to your model provider. Security reports go through GitHub private vulnerability reporting; no advisories are listed.
- Cost at scale
- Free; cost is model tokens and calls. Accessibility snapshots are text rather than images, and skeleton snapshots with drill-down keep large apps cheaper, but a long desktop task still means many observe and act calls. Hosted Vercel Sandbox VMs are billed by Vercel.
- Limits
- Desktop apps only, not web pages. Apps that publish no accessibility tree are invisible. Windows: unsigned binary, no headless type, and elevated apps need an elevated terminal. Linux: Ubuntu 24.04+ and a running AT-SPI bus. Local sandboxes are macOS only.
What’s good
- Snapshot-scoped refs fail rather than misclick
- Same command set on macOS, Windows and Linux
- CLI, MCP server and C library from one binary
- Per-platform docs on what works headless
- Local macOS VM sandbox for isolated runs
What’s not
- Can act in every app your account can open
- No per-app allow or block policy
- Windows binary unsigned; admin apps not drivable
- MCP mode shipped on 5 November 2026
Best for
Developers building agents that must operate native desktop apps with no API, who can run them on a spare machine or in the macOS VM sandbox.
agent-desktop alternatives
agent-desktop: common questions
- Is agent-desktop worth it?
- A well-built, widely used way to give an agent native desktop apps, with honest docs about what works headless on each OS, though it can act as you in every app you can open. We rate it 4.1 out of 5.
- How much does agent-desktop cost?
- Free and open source (Apache-2.0). Model calls are paid to your own provider; hosted Vercel Sandbox VMs are billed by Vercel.
- What are the downsides of agent-desktop?
- Can act in every app your account can open; no per-app allow or block policy; windows binary unsigned; admin apps not drivable; mCP mode shipped on 5 November 2026.
- Who is agent-desktop best for?
- Developers building agents that must operate native desktop apps with no API, who can run them on a spare machine or in the macOS VM sandbox.
- How much does agent-desktop cost for a team or in production?
- Free; cost is model tokens and calls. Accessibility snapshots are text rather than images, and skeleton snapshots with drill-down keep large apps cheaper, but a long desktop task still means many observe and act calls. Hosted Vercel Sandbox VMs are billed by Vercel.
- Is agent-desktop safe for company data?
- Runs locally; the MCP server speaks stdio and opens no port, though it opens a DevTools port on Chromium and Electron apps you launch for agent-browser. Snapshots, clipboard and screenshots can hold private content and go to your model provider. Security reports go through GitHub private vulnerability reporting; no advisories are listed.
- What are the best agent-desktop alternatives?
- The closest alternatives we have reviewed are Munim Computer Use and Playwright MCP.
Answers checked Nov 11, 2026.
Make agent-desktop? Show this rating on your site
