hermes-agent/website/docs/user-guide/features/computer-use.md

title, sidebar_position

title	sidebar_position
Computer Use	16

Computer Use

Hermes Agent can drive your desktop — clicking, typing, scrolling, dragging — in the background on macOS, Windows, and Linux. Your cursor doesn't move, keyboard focus doesn't change, and your virtual desktops / Spaces don't switch on you. You and the agent co-work on the same machine.

Unlike most computer-use integrations, this works with any tool-capable model — Claude, GPT, Gemini, or an open model on a local OpenAI-compatible endpoint. There's no Anthropic-native schema to worry about.

How it works

The built-in computer_use toolset is the recommended Hermes integration. It speaks MCP over stdio to cua-driver, an open-source background computer-use driver. Each platform uses the appropriate accessibility + input stack under the hood:

Platform	Accessibility tree	Input dispatch
macOS	AX (private SkyLight SPIs)	SLPSPostEventRecordTo — pid-scoped, no cursor warp
Windows	UIAutomation	SendInput + PostMessage — no focus steal
Linux	AT-SPI (X11 + Wayland)	XTest (X11) / virtual-keyboard (Wayland)

The result is the same on every platform: the agent can read the accessibility tree of any visible window AND post synthesized events without bringing it to front, switching virtual desktops, or moving the real OS cursor.

For the underlying contract — why background mode matters, the no-foreground invariant, click-dispatch internals — see cua.ai/docs/explanation/the-no-foreground-contract.

Which machine it drives

computer_use acts on the same machine the bot's screen lives on, never on the machine running Hermes Desktop. On a gateway with terminal.backend: local that is the gateway host. With a sandboxed terminal (docker, ssh, singularity) the driver runs inside the sandbox on the sandbox's own display, so it can only ever touch what the terminal can; the sandbox image must carry cua-driver (nousresearch/hermes-sandbox:desktop does). Modal, Daytona and Vercel sandboxes cannot host a display yet, so with those backends computer_use refuses unless bot_desktop.placement: gateway opts into driving the host. Details: Bot Screen → Where the screen runs.

Enabling

The driver ships with Hermes. cua-driver is pinned in pm/lock.json and is a default PM package: the installers, a bare hermes pm install, and hermes update install it on every macOS, Windows, and glibc Linux target (cua-driver publishes no musl or Android build). The desktop app's bundle carries it too. To leave it out, pass --skip-computer-use on POSIX or -SkipComputerUse on Windows (or run hermes pm install --without cua-driver); Hermes remembers the choice, and hermes pm install cua-driver undoes it.

If the download failed or you opted out earlier, any of these installs it:

• hermes tools → pick 🖱️ Computer Use — installs the driver automatically if it's still missing.
• Dashboard / desktop app → toggle the Computer Use toolset — if the driver is missing, the toggle kicks off the install in the background automatically (watch progress in the toolset panel).

Manual install / repair:

hermes computer-use install

This asks PM to prepare the pinned cua-driver package (verified against pm/lock.json) — it does not run the upstream installer. Use hermes computer-use status to verify the install.

Already have cua-driver? Hermes reuses it when it supports the 0.20 runtime contract. During setup, toolset enablement, hermes update, and the first computer_use call of a session, Hermes checks the local version and manifest. It repairs an old or incomplete standard installation through PM (at most once per session at runtime). A binary selected with HERMES_CUA_DRIVER_CMD stays under your control, so Hermes reports the incompatibility and leaves it unchanged.

If you install Cua Driver first, cua-driver skills install installs Cua's skill pack under ~/.cua-driver/skills/cua-driver. Hermes autodetection is a planned cua-driver follow-up, so currently point Hermes at that directory or symlink it into your skill space. You can also register raw Cua MCP tools as a custom MCP server, but that is an alternative for users who need the low-level interface. The built-in toolset provides Hermes actions, configuration, approvals, and diagnostics.

After installing, regardless of which path you took, grant the platform-appropriate prereqs:

Platform	Prereqs
macOS	System Settings → Privacy & Security → Accessibility + Screen Recording. Grant the identity named by hermes computer-use doctor (CuaDriver, com.trycua.driver, in every permission mode — the driver daemon always launches through CuaDriver.app).
Windows	None at install time. If you're driving over SSH (not RDP / console), you need the autostart pattern — see cua.ai/docs/how-to-guides/driver/windows-ssh for the Session 0 ↔ Session 1+ proxy.
Linux	A reachable display server: DISPLAY set for X11, or XDG_SESSION_TYPE=wayland. Wayland sessions need an XWayland bridge for capture. AT-SPI must be on (default on GNOME/KDE/Xfce).

Then start a session with the toolset enabled:

hermes -t computer_use chat

or add computer_use to your enabled toolsets in ~/.hermes/config.yaml.

Permission modes and logged-in browser profiles

Hermes maps its existing approval UX onto cua-driver's immutable runtime modes. Permission mode and capability manifest approval are launch settings. They cannot change after the runtime starts:

Hermes session	cua-driver mode	Human intervention
Manual or smart approvals (default)	standard	Normal Hermes approvals; Cua stops at its protected boundary
computer_use.permission_mode: bounded + reviewed manifest	private bounded daemon	You review and approve the capability manifest once, at launch
--yolo, /yolo, or approvals.mode: off	private unrestricted daemon	One explicit Hermes risk acceptance; no runtime Cua prompts

Browser work — including pages in a signed-in profile — goes through the browser toolset (browser_exec), not computer_use. The former computer_use.grant_existing_profile opt-in was removed along with the typed browser route; a leftover key in config.yaml is ignored.

Bounded mode for repeatable automation

For recurring browser automation (cron jobs, scheduled research against an authenticated app), bounded mode uses a capability manifest you review once:

# config.yaml
computer_use:
  permission_mode: bounded
  capability_manifest: ~/.hermes/cua-manifest.yaml

The manifest names the apps, browser profile kinds, allowed origins, and typed tools the session may use (see the cua-driver permission modes reference for the format). Hermes launches a private runtime with --capability-manifest ... --approve-capability-manifest; anything outside the manifest fails closed inside cua-driver. A missing or unreadable manifest fails loudly at session start rather than silently downgrading. Session YOLO still overrides bounded for that one session.

On macOS, private-session daemons launch through the installed CuaDriver.app bundle (so permission grants attribute to the driver's own identity instead of resetting with every Hermes build), and Hermes verifies the bundle's code signature — exact com.trycua.driver identifier and the official signing team — before launching it. If you build cua-driver from source (unsigned), opt in explicitly:

# config.yaml
computer_use:
  allow_unsigned_driver: true   # local driver development only

Each MCP transport owns a private lifecycle session inside its runtime. A public session name is only a label for cursor identity and session-scoped state. It does not select, share, or keep a runtime alive. Turning /yolo off, resetting or closing the Hermes session, cancellation cleanup, or process exit closes that transport session. Hermes also stops private runtimes that it launched for bounded or unrestricted access. One Hermes conversation cannot change another runtime's mode or grants. Bounded and unrestricted modes use a private embedded daemon, launched through CuaDriver.app on macOS (see above).

smart approval remains standard: an LLM classification cannot stand in for a reviewed manifest.

YOLO/unrestricted mode does not protect against prompt injection or unintended input. Use it only in a disposable VM or with accounts and data whose full compromise you accept.

hermes computer-use doctor — your first triage stop

hermes computer-use doctor runs cua-driver's structured health_report MCP tool and prints a per-check matrix. It's the single fastest way to find out why an action isn't working.

$ hermes computer-use doctor
⚠️  cua-driver VERSION on darwin: degraded
  ✅ binary_version: cua-driver VERSION
  ✅ platform_supported: macOS 26.4.1 (arm64)
  ✅ session_active: MCP session is active.
  ❌ bundle_identity: Process has no CFBundleIdentifier.
      → Run the binary inside CuaDriver.app so TCC grants attribute correctly.
  ✅ tcc_accessibility: Accessibility is granted.
  ✅ tcc_screen_recording: Screen Recording is granted.
  ✅ ax_capability: AX is trusted and reachable.
  ✅ screen_capture_capability: ScreenCaptureKit reachable; 1 display(s) shareable.

• Exit code 0 when overall is ok — everything's wired up.
• Exit code 1 when degraded or failed — at least one check failed; the hint on each failure tells you what to fix.
• Exit code 2 when the cua-driver binary itself isn't reachable.

Useful flags:

• --include CHECK — run only the listed checks (repeat for multiple)
• --skip CHECK — skip a check (wins over --include)
• --json — emit the raw structured payload, same shape as the tools/call health_report MCP response

The check matrix is platform-aware: bundle_identity / tcc_* are skip on Windows + Linux because those concepts don't apply. ax_capability checks AX on macOS, UIA on Windows, AT-SPI on Linux — each with the right diagnostic hint when it can't reach.

On Linux, where the daemon is a hand-written systemd user unit or XDG autostart entry rather than a managed autostart, doctor also reads those units: a cua-driver ExecStart pointing at a pruned packages/releases/<version>/ directory is reported as a failing daemon unit (...) check (point it at ~/.cua-driver/packages/current/cua-driver), and a unit that runs cua-driver serve gets a daemon (...) check that connects to its socket — fail when nothing is listening (crash loop, stopped, never started), pass when the daemon answers. Reinstalling the driver does not start a daemon; systemctl --user status <unit> does. hermes computer-use status prints the same dead-daemon line and exits 1.

The agent cursor and sessions

When the agent acts, you'll see a tinted overlay cursor glide across the screen to where each click / type / scroll lands. The real OS cursor never moves. The overlay shows where the agent is acting. Each Hermes run declares a public cua-driver session name (something like hermes-3a7b9c14d2e8). The name labels cursor identity and related state, so concurrent runs and subagents get distinct cursors. The MCP transport owns the private lifecycle session inside the runtime; the public name does not.

The overlay cursor is cosmetic — captures, clicks, and typing all work without it. Hermes disables it automatically where it is a known failure mode: macOS (idle CPU burn), headless Linux / WSL2 / containers, and Linux X11 desktops (the overlay is a fullscreen always-on-top window that can get stuck over every workspace after an unclean session end, wedging desktop input). Linux Wayland and Windows keep the overlay. Set computer_use.no_overlay: false in config.yaml to force the cursor on (or true to force it off) on any platform.

Tune the cursor with cua-driver's CLI flags or the runtime set_agent_cursor_style MCP tool — see cua.ai/docs/how-to-guides/driver/personalize-cursor for the full menu (built-in arrow vs teardrop silhouette, custom SVG / PNG / ICO via --cursor-icon, runtime gradient colors, bloom halo).

Going deeper — the cua-driver skill pack

Hermes keeps its wrapper skill (skills/autonomous-ai-agents/computer-use/SKILL.md) focused on the Hermes-side computer_use workflow and action vocabulary. For platform details, recording semantics, browser page interaction, and other deep Cua behavior, install the skill pack that the cua-driver team ships and maintains directly:

cua-driver skills install

The command links the pack into ~/.hermes/skills/cua-driver (Hermes is one of the agents cua-driver skills status reports). The wrapper remains the workflow layer: the pack documents the driver's own MCP vocabulary (get_window_state, element_token, snapshot_id), which the computer_use wrapper translates to for you — keep calling computer_use(action=...). The pack contains:

File	Topic
SKILL.md	The cross-platform core (snapshot invariant, no-foreground contract, click dispatch, AX-tree mechanics)
MACOS.md	macOS specifics: no-foreground contract, AXMenuBar navigation, SkyLight click dispatch, Apple Events JS bridge
WINDOWS.md	Windows specifics: UIA tree, UWP / ApplicationFrameHost hosting, Session 0 isolation, autostart pattern
LINUX.md	Linux specifics: AT-SPI tree, X11 / Wayland, terminal-emulator detection
RECORDING.md	Trajectory + video recording semantics
WEB_APPS.md	Browser-page interaction tips
TESTS.md	Replay-by-trajectory workflow

These are platform deep dives, not duplicates of the Hermes skill — when an agent reports "on Windows, my click landed on the wrong element," it reads WINDOWS.md for the UIA / UWP context that explains why and what to do differently.

cua-driver skills status shows what's installed and which agent harnesses it's linked into. Today the autodetect list covers Claude Code, Codex, OpenCode, OpenClaw, and Antigravity; Hermes autodetection is planned as a follow-up in trycua/cua — until then, run cua-driver skills install once and point your harness at the resulting ~/.cua-driver/skills/cua-driver directory (or symlink it into your usual skill space).

Quick example

User prompt: "Find my latest email from Stripe and summarise what they want me to do."

The agent's plan (this is the same shape on macOS / Windows / Linux — the model substitutes the platform's idiomatic shortcut and app name):

1. computer_use(action="capture", mode="som", app="Mail") — gets a screenshot of the email app with every sidebar item, toolbar button, and message row numbered.
2. computer_use(action="click", element=14) — clicks the search field.
3. computer_use(action="type", text="from:stripe")
4. computer_use(action="key", keys="return", capture_after=True) — submit and get the new screenshot.
5. Click the top result, read the body, summarise.

During all of this, your cursor stays wherever you left it and the email app never comes to front.

Receiving the actual screenshot

Screenshots taken during computer control are normally internal — they exist so the model can see the screen, and the agent replies in text. But every image capture also saves a bounded, shareable copy under Hermes' image cache and reports its path, so on attachment-capable surfaces (Telegram, Discord, Desktop, and other gateway platforms) you can simply ask:

"Send me a screenshot of my screen."

and the agent delivers the real image as a native attachment, not just a description. On the CLI there is no attachment channel, so the agent gives you the saved file's path instead.

Only the 20 most recent capture files are kept, and screenshots are never sent automatically — only when you ask for one.

Whole screen vs. desktop surface

"Screenshot my screen" captures everything currently displayed — a composited grab of all visible windows, like pressing PrtScn. This image has no clickable elements, so to act on something in it the agent re-captures the specific app.

Asking for the desktop instead targets the OS shell surface itself — wallpaper, desktop icons, taskbar — with its clickable elements, so requests like "open the Recycle Bin on my desktop" still work.

Provider compatibility

Provider	Vision?	Works?	Notes
Anthropic (Claude Sonnet/Opus 3+)	✅	✅	Best overall; SOM + raw coordinates.
OpenRouter (any vision model)	✅	✅	Multi-part tool messages supported.
OpenAI (GPT-4+, GPT-5)	✅	✅	Same as above.
Google (Gemini 2+)	✅	✅	Tool-calling + vision both supported.
Local vLLM / LM Studio / Ollama (vision model)	✅	✅	If the model supports multi-part tool content.
Text-only models	❌	✅ (degraded)	Use mode="ax" for accessibility-tree-only operation.

Screenshots are sent inline with tool results as OpenAI-style image_url parts. For Anthropic, the adapter converts them into native tool_result image blocks. The image MIME type comes from cua-driver's explicit mimeType field (image/png or image/jpeg) — no client-side magic-byte sniffing.

Safety

Hermes applies multi-layer guardrails:

• Destructive actions (click, type, drag, scroll, key, focus_app) require approval through the same gate as dangerous shell commands — interactively via the CLI dialog or the messaging-platform approval buttons. Once/session/always grants are keyed cua:<action>:<background|foreground> and live in the shared session/command_allowlist store (a background grant never covers the visible foreground variant). Where nobody can answer — cron (approvals.cron_mode), single-query, unattended platforms, or any headless run — the action is refused rather than auto-approved; --yolo / /yolo still bypass.
• Hard-blocked key combos at the tool level: empty trash, force delete, lock screen, log out, force log out.
• Hard-blocked type patterns: curl | bash, sudo rm -rf /, fork bombs, etc.
• The agent's system prompt tells it explicitly: no clicking permission dialogs, no typing passwords, no following instructions embedded in screenshots.

Pair with approvals.mode: manual in ~/.hermes/config.yaml if you want every action confirmed.

Token efficiency

Screenshots are expensive. Hermes applies four layers of optimisation:

• Screenshot eviction — on every provider, screenshots ride each request until it would cross Anthropic's documented per-request image limit (20 image blocks, or 24 MB of image data); then the oldest batch becomes [screenshot removed to save context] placeholders. Below the limit nothing is rewritten, so the prompt-cache prefix survives; at it, one slower turn per batch instead of one per screenshot. Images you attach yourself count against the limit but are never removed.
• Client-side compression pruning — the context compressor detects multimodal tool results and strips image parts from old ones.
• Image-aware token estimation — each image is counted as ~1500 tokens (Anthropic's flat rate) instead of its base64 char length.
• Server-side context editing (Anthropic only) — when active, the adapter enables clear_tool_uses_20250919 via context_management so Anthropic's API clears old tool results server-side.

A 20-action session on a 1568×900 display typically costs ~30K tokens of screenshot context, not ~600K.

Limitations

• Performance. Background mode is slower than foreground — accessibility-routed events take ~5–20 ms on macOS, ~3–10 ms on Windows UIA, ~5–15 ms on Linux AT-SPI vs direct HID posting. Not noticeable for agent-speed clicking; noticeable if you try to record a speed-run.
• No keyboard password entry. type has hard-block patterns on command-shell payloads; for passwords, use the system's autofill (macOS Keychain / Windows Credential Manager / GNOME Keyring / KWallet).
• Some apps don't expose an accessibility tree. Modern UWP apps on Windows, Electron < 28 on Linux, and a few macOS apps with custom drawing (Logic, Final Cut, some games) have sparse or empty AX trees. Fall back to pixel coordinates if the tree is empty — or skip the task entirely.
• Windows: elevated (admin) windows can't be driven from a normal agent. Windows UIPI (User Interface Privilege Isolation) enforces integrity-level boundaries: a Medium-integrity process (the default Hermes agent) cannot enumerate the UIA tree of, or inject mouse input into, a window owned by a High-integrity (Administrator) process. Symptom: capture(mode='som') returns 0 elements and click(...) reports success while doing nothing, even though the screenshot renders fine (GDI capture sits below the integrity check). Keyboard events partially bypass UIPI, so Tab / Enter can still navigate an elevated dialog. This is an OS constraint, not a cua-driver bug — it affects every Windows automation stack. To drive elevated windows, run the Hermes agent itself at High integrity (launch from an elevated terminal); otherwise target non-elevated windows.
• Windows: hermes computer-use doctor fails with "Access is denied" while the tool works. A cua-driver installed under C:\Program Files\WindowsApps cannot be executed by the Hermes venv interpreter (WinError 5 from CreateProcess), even though the shell resolves the same binary fine. The doctor now reports this as a diagnosis instead of a traceback. Fix once: reinstall with the upstream installer (lands under your user profile) or set HERMES_CUA_DRIVER_CMD to a copy outside WindowsApps.
• Platform-specific deployment gotchas:
  • macOS uses private SkyLight SPIs. Apple can change them in any OS update. Hermes warns when the installed cua-driver is older than the version it was tested against.
  • Windows SSH sessions run in Session 0, which has no interactive desktop. Drive Hermes from inside the RDP / console session, or set up cua-driver's autostart Scheduled Task — windows-ssh has the recipe.
  • Linux requires a reachable display server. Headless servers get one from Bot Screen: a per-profile Xfce desktop over TigerVNC, streamed into Hermes Desktop, where you can take over for logins and 2FA. You start it from the Desktop's Screen pane or hermes computer-use screen start; it starts on first use (the first computer_use call or headed browser use) only when bot_desktop.auto_start: true is set (off by default). Pure Wayland sessions need an XWayland bridge for screen capture (cua-driver's Wayland inject path handles input independently).

For cross-platform GUI automation without the desktop overhead (and without TCC / Session 0 / X11 setup), the browser toolset uses a real headless Chromium and is the right answer for web-only tasks.

Configuration

Permission mode and manifest (see Permission modes above):

computer_use:
  permission_mode: standard        # standard (default) | bounded
  capability_manifest: ""          # capability manifest path, required for bounded

On Linux, native Wayland support remains an explicit opt-in. Hermes passes the opt-in to every cua-driver process, including gateway sessions, only when that process also has WAYLAND_DISPLAY:

computer_use:
  native_wayland: true

Restart a running gateway after changing this setting.

Override the driver binary path (tests / CI / local builds):

HERMES_CUA_DRIVER_CMD=/path/to/your/cua-driver

Windows auto-start (opt-in)

On Windows, cua-driver can run from a per-boot Scheduled Task (cua-driver-serve) so it is already listening when Hermes needs it. This task is opt-in: by default Computer Use starts the driver on demand, per session — exactly as on macOS and Linux — and no scheduled task is registered when you install or enable the toolset (#97389).

Set this in config.yaml to opt in (the task is registered — or repaired — the next time the driver is installed or the toolset is enabled):

computer_use:
  autostart: true   # default: false (on-demand; no scheduled task)

You need this when driving Windows over SSH: Session 0 has no interactive desktop, so an on-demand driver cannot reach one (windows-ssh has the recipe). If the task exists but you want it gone, remove it with cua-driver autostart disable (or schtasks /Delete /TN cua-driver-serve) from an elevated shell — Hermes does not re-register it once computer_use.autostart is false.

Swap the backend entirely (for testing):

HERMES_COMPUTER_USE_BACKEND=noop   # records calls, no side effects

Telemetry

cua-driver ships with anonymous usage telemetry (PostHog) enabled by default upstream. Hermes disables it for you — on every cua-driver invocation (the MCP backend, status, doctor, and install) Hermes sets CUA_DRIVER_RS_TELEMETRY_ENABLED=0 in the driver's environment.

To opt back in (let cua-driver use its own default and send telemetry), set this in config.yaml:

computer_use:
  cua_telemetry: true   # default: false (telemetry off)

When it's on, hermes computer-use doctor reports telemetry: enabled; when off (the default), it reports telemetry: disabled via CUA_DRIVER_RS_TELEMETRY_ENABLED.

Testing against a local cua-driver build

When you're developing cua-driver itself — or want to test an unreleased fix — point Hermes at a binary you built from source instead of the published release. Hermes resolves the driver with shutil.which("cua-driver") and does not enforce HERMES_CUA_DRIVER_VERSION, so a local build (reported as 0.0.0-local-*) is accepted as-is. Two approaches:

Option A — install-local (build + put it on PATH)

From your trycua/cua checkout, run the upstream local installer. It builds the Rust backend in release mode and drops cua-driver into the same install layout the production installer uses, adding its bin dir to your PATH:

# Windows (PowerShell), from the cua repo root
./libs/cua-driver/scripts/install-local.ps1 -NoAutoStart

# macOS / Linux, from the cua repo root  (defaults to a debug build without --release)
./libs/cua-driver/scripts/install-local.sh --release

• Windows stages the build under %USERPROFILE%\.cua-driver\packages\… and junctions %LOCALAPPDATA%\Programs\Cua\cua-driver\bin (added to your User PATH) to it. macOS/Linux symlinks cua-driver into ~/.local/bin (override with --bin-dir <path>).
• -NoAutoStart skips registering the cua-driver-serve logon daemon — you don't need it for Hermes testing (see notes).

Then open a fresh shell (so the PATH change is visible) and confirm:

cua-driver --version                 # local builds report 0.0.0-local-release
# Windows:      (Get-Command cua-driver).Source
# macOS/Linux:  which cua-driver

Option B — point Hermes straight at the built binary (fastest loop)

Skip the install ceremony entirely: cargo build and set HERMES_CUA_DRIVER_CMD to the resulting binary. Best for rapid edit/build/test.

cargo build -p cua-driver            # add --release for a release build; run from libs/cua-driver/rust

# Windows (.env)
HERMES_CUA_DRIVER_CMD=C:\path\to\cua\libs\cua-driver\rust\target\debug\cua-driver.exe
# macOS / Linux (.env)
HERMES_CUA_DRIVER_CMD=/path/to/cua/libs/cua-driver/rust/target/debug/cua-driver

Confirm Hermes is using your build

• hermes computer-use status prints the resolved binary path and version.
• hermes computer-use doctor confirms the binary is reachable and exercises the full MCP path end-to-end.
• In a session, computer_use(action="capture") exercises the spawned cua-driver mcp child process.

Notes & gotchas

• Hermes spawns a cua-driver mcp stdio proxy. In a normal session the proxy connects to (and may start) the standard machine daemon. In explicit Hermes YOLO, Hermes instead owns a private cua-driver serve --embedded child and points the proxy at its private socket or named pipe. The Windows autostart/UIAccess pattern still matters for interactive Session 1+ input from SSH — see the Limitations section.
• Locked binary on Windows. A running cua-driver-serve daemon can hold cua-driver.exe and block an overwrite on rebuild. install-local.ps1 renames the locked binary out of the way automatically; if you cargo build manually (Option B), stop it first with cua-driver autostart disable (or schtasks /End /TN cua-driver-serve).
• Rebuild loop. After editing cua-driver source, re-run install-local (rebuilds, restages, flips the current junction) for Option A, or just re-cargo build for Option B — no Hermes change needed either way.
• Local builds skip the version check. Hermes warns when the installed cua-driver is older than its per-OS tested baseline, but exempts 0.0.0-local-* dev builds — so your local build never triggers that warning.

Troubleshooting

First action when anything's off: run hermes computer-use doctor. The structured per-check matrix tells you (and any agent helping you debug) exactly what's wrong.

Specific failure modes the doctor doesn't catch:

computer_use backend unavailable: cua-driver is not installed — Run hermes computer-use install to fetch the cua-driver binary, or run hermes tools and enable the Computer Use toolset.

Clicks seem to have no effect — Capture and verify. A modal you didn't see may be blocking input. Dismiss it with escape or the close button.

macOS: System Settings shows CuaDriver ON, but hermes computer-use permissions status / doctor report Accessibility or Screen Recording as not granted — the stored grant is stale. macOS keys each permission row to the app's code-signing requirement; a row written for an earlier CuaDriver build stops matching after a driver update, and flipping the toggle does not rewrite it. Reset the affected rows and re-grant:

tccutil reset Accessibility com.trycua.driver
tccutil reset ScreenCapture com.trycua.driver
hermes computer-use permissions grant

Element indices are stale — SOM indices are only valid until the next capture. Re-capture after any state-changing action. The wrapper carries opaque element_tokens for stale detection — you'll see an explicit error rather than a wrong click.

"blocked pattern in type text" — The text you tried to type matches the dangerous-shell-pattern list. Break the command up or reconsider.

Empty captures on Linux — DISPLAY not set, or you're on pure Wayland without an XWayland bridge. hermes computer-use doctor will flag this as ax_capability: fail with a Set DISPLAY (X11)… hint.

Empty captures on Windows over SSH — You're in Session 0 (the services session). Drive from RDP / console directly, or set up the autostart pattern — see cua.ai/docs/how-to-guides/driver/windows-ssh.

See also

• Hermes-side skill — skills/autonomous-ai-agents/computer-use/SKILL.md — teaches the Hermes computer_use action vocabulary; this is what the agent loads.
• cua-driver skill pack — for platform-specific deep dives (macOS no-foreground contract, Windows UIA + Session 0, Linux AT-SPI
  • X11/Wayland, recording, browser pages), run cua-driver skills install and read MACOS.md / WINDOWS.md / LINUX.md / RECORDING.md / WEB_APPS.md. Hermes autodetection is a planned follow-up; currently point Hermes at the installed pack directory or symlink it into your skill space.
• cua.ai/docs — the cua-driver project's documentation:
  • What is computer use? — concept intro
  • The no-foreground contract — why background mode matters
  • Install reference — cross-platform install details
  • Personalize the agent cursor — built-in shapes, custom assets, runtime overrides
  • Drive Windows over SSH — the Session 0 → Session 1+ autostart pattern
  • Keep cua-driver running — autostart / daemon lifecycle
  • Connect your agent — register cua-driver with various harnesses (Hermes among them)
• cua-driver source (trycua/cua)
• Browser automation for cross-platform web tasks where you don't need to drive native apps.